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Application Number: 09/987,59Wv .& ) Examiner Kumiko C. KOYAMA 

Filed: November 15, 2001 ) Confirmation No. 7385 



In re Application Of: 



Ajit RAJASEKHARAN fe M 1 5 ffl* J ) Group Art Unit: 2876 




For: SYSTEM AND METHOD FOR ) 

AUTHORING AND PROVIDING ) 

INFORMATION RELEVANT TO A ) 

PHYSICAL WORLD ) 

REPLY TO NOTICE REGARDING CHANGE OF POWER OF ATTORNEY 

U.S. Patent and Trademark Office 
2011 South Clark Place 
Customer Window 

Crystal Plaza Two, Lobby, Room 1B03 s 
Arlington, VA 22202 



The United States Patent & Trademark Office (PTO) has mistakenly acknowledged an 
improper revocation of Applicant's appointed representatives, Le. 9 Rodger Tate and Trevor 
Coddington of Hunton & Williams LLP ("Tate et aL"), as indicated in the Notice Regarding 
Change of Power of Attorney issued on June 7, 2004. Sole Applicant and inventor in the instant 
application, Mr. Rajasekharan, maintains that the appointment of Tate et aL as his 
representatives has not been revoked. Accordingly, it is respectfully requested that the PTO: (1) 
update all appropriate records to reflect that Tate et aL are Mr. Rajasekharan' s properly 
appointed representatives; (2) notify Mr. Rajasekharan and/or Tate et aL that the PTO correctly 
recognizes Tate et aL as representatives able to prosecute the instant application and not non- 
party Ms. Kovesdi or her representatives at Jones Day; and (3) in the event that any office action 
has been issued and incorrectly sent to Jones Day, mail Tate et aL a copy of such office action 
with the period for response appropriately restarted. 

The inventorship of the instant application was set forth at the time of filing in a properly 
executed Declaration and Power of Attorney signed by the inventor of record, Mr. Rajasekharan, 



Dear Sir: 
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pursuant to 35 U.S.C. § 115-116 and 37 C.F.R. § 1.63. Mr. Rajasekharan appointed, inter alia, 
Rodger Tate and Trevor Coddington of Hunton & Williams LLP (formerly with Brobeck, 
Phleger & Harrison LLP) as his representatives to conduct prosecution of the instant application 
before the PTO. This appointment has not been revoked in any way by Mr. Rajasekharan (nor 
Assignee Readia, Inc.). 

Apparently in response to non-party Ms. Kovesdi' s attempt to submit a Revocation and 
Power of Attorney on May 24, 2004, in which she revokes "any and all previous powers," on the 
deceitful representation that she is an "Applicant of Record" when in fact she is not, the PTO by 
mistake revoked Mr. Rajasekharan's appointed representatives. See Notice Regarding Change of 
Power of Attorney issued on June 7, 2004. Ms. Kovesdi is not an inventor or applicant of record, 
and the PTO itself has previously ruled that Ms. Kovesdi is not a property in interest in the 
instant application. See PTO Letter of January 12, 2004. Thus, Ms. Kovesdi has no power 
whatsoever to revoke Mr. Rajasekharan's appointed representation. 

No fee is believed to be necessary for the PTO's consideration of this Reply. In the event 
that the U.S. Patent and Trademark Office considers this Reply a Petition under 37 C.F.R. § 
1.182, Mr. Rajasekharan submits that no petition fee is required because the reason for filing 
such a Reply/Petition is due to PTO mistake. Nonetheless, if a fee is otherwise deemed 
necessary by the PTO to consider this Reply or to maintain the instant application pending, 
please charge such fee to the undersigned's Deposit Account No. 50-0206. 

Respectfully submitted, 



Dated: June 15, 2004 




Hunton & Williams LLP Registration No. 46,633 

Intellectual Property Department 

1900 K Street, N.W., Suite 1200 

Washington, DC 20006-1 109 

(202) 955-1500 (telephone) 

(202) 778-2201 (facsimile) 



63044.000002 WASHINGTON 427052vl 
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IN THE UNITED STATES PATENT AND TRADEMARK OFFICE 
In re Application of: 



Ajit RAJASEKHARAN g Jl « 1 5 2004 l, 

Application Number: 09/987,59*^ 
Filed: March 25, 2002 



Group Art Unit: 2876 
Examiner K. KOYAMA 
Confirmation No. 7385 



For: SYSTEM AND METHOD FOR 

AUTHORING AND PROVIDING 
INFORMATION RELEVANT TO A 
PHYSICAL WORLD 

MR. RAJASEKHARAN'S REPLY TO 
MS. KOVESDPS REQUEST FOR RECOGNITION AS 
A PROPER PARTY UNDER 37 C .F.R. §§ 1.181-1.183 

U.S. Patent and Trademark Office 
2011 South Clark Place 
Customer Window 

Crystal Plaza Two, Lobby, Room 1B03 
Arlington, VA 22202 

Dear Sir: 

Applicant, Mr. Ajit Rajasekharan, of the above-captioned application respectfully 
requests the Commissioner to consider the following remarks in view of the Request for 
Recognition as a Proper Party under 37 C.F.R. §§ 1.181-1.183 submitted by non-party Ms. 
Kovesdi on May 24, 2004 (the "Kovesdi Petition"). Particularly, Ms. Kovesdi has requested that 
the United States Patent & Trademark Office (PTO) recognize her as a party to the instant 
application or in the alternative, to prevent any patent from issuing before the inventorship 
dispute between Ms. Kovesdi and Mr. Rajasekharan is resolved on the merits. Although 
Applicant ultimately seeks an equitable judgment to Ms. Kovesdi' s disputed claim of joint 
inventorship, Applicant respectfully submits that the Kovesdi Petition is frivolous and the relief 
sought therein should be dismissed on any one of the following grounds: 

(1) Ms. Kovesdi lacks standing to ask for the relief sought as she is not recognized as a 
proper party in interest to the instant application; 

(2) Ms. Kovesdi's premise for recognizing her as a proper party in interest is frivolous; 



Page 1 of 13 



Application No. 09/987,597 

Reply to Kovesdi Petition of May 24, 2004 

(3) Ms. Kovesdi has failed to show good cause for the PTO granting her extraordinary 
and inappropriate request; and 

(4) the disputed claim of joint inventorship is not ripe for the PTO's consideration. 
I. RELEVANT HISTORY 

The relevant procedural history of this application and the applications related thereto, 
and the circumstances surrounding the dispute with Ms. Kovesdi are summarized as follows. 

At the outset, it is worthy to note that the instant application claims priority to U.S. Patent 
Provisional Application No. 60/306,356 ("the '356 Provisional"), filed on July 18, 2001. 

A. The Joint '356 Provisional 

The inventive entity designated in the '356 Provisional filed on July 18, 2001, consists of 
Mr. Rajasekharan and Ms. Kovesdi. 

The '356 Provisional recites seventy claims. 1 An objective review of the facts as 
evidenced by his recollections and records led Mr. Rajasekharan to the position that only two 
dependent claims, Le., claims numbered 44 and 45, found in the '356 Provisional were jointly 
conceived with Ms. Kovesdi; all sixty-eight other claims being solely conceived by Mr. 
Rajasekharan. This position has been consistently maintained by Mr. Rajasekharan throughout 
the course of his interaction with Ms. Kovesdi and the PTO. 

Around the time of filing of the '356 Provisional, the business relationship between Ms. 
Kovesdi and Mr. Rajasekharan failed and all direct communications among them ceased. 

B. The Instant '597 Application 

Mr. Rajasekharan filed the instant application on November 15, 2001, within four months 
of the filing date of the '356 Provisional. Although the instant application is based upon and 
claims priority to the '356 Provisional, it does not include the subject matter of claims 44 and 45 
originally found in the '356 Provisional nor any other claimed subject matter to be jointly 
conceived with Ms. Kovesdi. Therefore, the instant application designates Mr. Rajasekharan as 
a sole inventor of the claimed subject matter rather than a joint inventor with Ms. Kovesdi as 
designated in the '356 Provisional. The instant application also omits certain disclosure found in 
the '356 Provisional as well as adds new disclosure and claims not found in the '356 Provisional. 



1 Although the last claim is numbered claim 71, only seventy claims are provided since claim 22 
is omitted. 
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A marked-up version of the instant application showing all modifications vis-^-vis the '356 
Provisional is attached as Appendix A. 

An Office Action issued on March 27, 2003, which rejected all pending claims, i.e M 
claims 1-70, based upon prior art cited by the Examiner. Mr. Rajasekharan timely submitted a 
Response to the Office Action on June 11, 2003. Apparently as a result of Ms. Kovesdi's ill- 
advised attempt to intervene in the prosecution of the instant application (in which she is not a 
party) by way of a Petition submitted on April 21, 2003, which was not considered by the PTO 
as indicated in a Letter of January 12, 2004 {see Kovesdi Petition, Exhibit R), Mr. Rajasekharan 
to this day still awaits consideration of his June 11 th Response submitted one year earlier. 

Ms. Kovesdi again attempts join the ex parte prosecution of the instant application by 
way of her present Petition submitted on May 24, 2004. Such a request blatantly contravenes the 
PTO's prior holding in the Letter of January 12 th , which expressly states that petitions by those 
other than a proper party in interest will not be treated on the merits. Moreover, Ms. Kovesdi is 
seemingly intent on disrupting the prosecution of the instant application by any means possible 
whether inappropriate or not, as evidenced by her Revocation and Power of Attorney 
concurrently submitted with her Petition. In an apparent attempt to improperly take control of 
the instant prosecution away from Mr. Rajasekharan, her Revocation revokes "any and all 
previous powers" on the deceitful representation that she is an "Applicant of Record" when in 
fact, she is not (Ms. Kovesdi is not a property in interest and has been ruled as such by the PTO 
as noted above). In response thereto, the PTO has mistakenly revoked the appointment of Mr. 
Rajasekharan' s representatives, i.e., Rodger Tate and Trevor Coddington of Hunton & Williams 
LLP, as noted in the Notice Regarding Change of Power of Attorney issued on June 7, 2004. 
Applicant, Mr. Rajasekharan, submits that he has not revoked the appointment of Rodger Tate 
and Trevor Coddington as his representatives and is supporting such by way of documentation 
concurrently submitted herewith. 

Ms. Kovesdi also filed a Petition under 37 C.F.R. §§ 1.181-1.183 on April 22, 2004, in 
U.S. Patent No. 10/103,777 ("the '777 Application") (to which she is not a party) to request the 
same relief as that sought in the instant petition. 2 



2 The 777 Application, filed on March 25, 2002, which is a continuation-in-part of the instant 
application, names Mr. Rajasekharan along with Mr. Millman as a joint inventive entity. Ms. 
Kovesdi was not joined as a party, Le. y Applicant, to the '777 Application. 
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C. Ms. Kovesdi's '952 Joint Application 

After the filing of the instant application, Ms. Kovesdi filed U.S. Patent Application No. 
10/035,952 ("the '952 Application") on December 26, 2001, which is based upon and identical 
to the '356 Provisional. The '952 Application names Mr. Rajasekharan and Ms. Kovesdi as a 
joint inventive entity. 

It was not until April 22, 2002, that the '952 Application was brought to the attention of 
Mr. Rajasekharan. See Appendix B. At the request of Ms. Kovesdi's representative, Gary 
Jarosik 3 , and to reply to the Notice of Missing Parts issued on April 4, 2002, forwarded at that 
time, Mr. Rajasekharan timely submitted directly to the PTO a Declaration on June 4, 2002, 
declaring himself as a joint inventor "of the subject matter to which at least one claim is 
directed" due to the recognition that the '952 Application includes claimed subject matter 
corresponding to the two jointly conceived dependent claims 44 and 45 found in the '356 
Provisional. 4 Because of the adverse relationship that had developed between the parties and to 
protect Mr. Rajasekharan's interests in the '952 Application, a Petition under 37 C.F.R. § 1.182 
was concurrently submitted therewith requesting that Mr. Rajasekharan formally join the 
prosecution in the '952 Application under the dual representation provisions of MPEP § 402.10, 
thereby permitting Mr. Rajasekharan to have his own independent representation vis-a-vis Mr. 
Kovesdi. See Kovesdi Petition, Exhibit G. Due to the unintentional mistake of omitting Ms. 
Kovesdi in the inventive entity designated, the PTO found Mr. Rajasekharan's Declaration 
defective and therefore, dismissed his Petition as moot. See Kovesdi Petition, Exhibit H. On 
August 23, 2002, Mr. Rajasekharan resubmitted his Declaration, but with Ms. Kovesdi properly 
identified as the other joint inventor, along with a substantively identical Petition under 37 
C.F.R. § 1.182 as previously submitted In the meantime, Ms. Kovesdi submitted on May 1, 
2002, a Petition under 37 C.F.R. § 1.47(a) asking to recognize Mr. Rajasekharan as a non- 



3 For reasons unknown, Mr. Jarosik later removed himself as counsel for Ms. Kovesdi sometime 
in or around February of 2003. See Appendix C. 

4 Claims 44 and 45 of the '356 Provisional have been renumbered claims 43 and 44 by the 
Examiner in the '952 Application since claim 22 was mistakenly omitted. 
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signing inventor and to prosecute the application on behalf of both parties. 5 See Kovesdi 
Petition, Exhibit F. 

Mr. Rajasekharan's Petition under 37 C.F.R. § 1.182 was granted and Ms. Kovesdi's 
Petition under 37 C.F.R. § 1.47(a) was deemed moot by way of the PTO's Decision of 
September 17, 2002, which expressly provided that: (1) Mr. Rajasekharan's Declaration of 
August 23, 2002, is compliant with 37 C.F.R. § 1.63 and the '952 Application does not have any 
rule 1.47(a) status, and (2) all further correspondence to the PTO must be signed by a 
representative of each party in accordance with MPEP § 402.10 to assure that all interests are 
properly and effectively represented. See Kovesdi Petition, Exhibit I. 

On April 16, 2003, Mr. Rajasekharan's representatives received a copy of a first Office 
Action via Ms. Kovesdi' s representatives, which was originally mailed by the PTO nearly one 
month before on March 18, 2003. See Appendix D. The Office Action rejected all claims in the 
'952 Application in view of prior art raised by the Examiner. See Kovesdi Petition, Exhibit J. 
Mr. Rajasekharan formulated and forwarded to Ms. Kovesdi' s representatives a ready-to-file 
Response to the Office Action on May 21, 2003, which addressed the prior art rejections and 
canceled all claims that we maintained as solely conceived by Mr. Rajasekharan since those 
claims were being pursued in the instant application. See Kovesdi Petition, Exhibit L. By 
forwarding Mr. Rajasekharan's proposed Response approximately two months after the Office 
Action's date of mailing, nearly four full months remained in the statutory response period for 
Ms. Kovesdi to review such and provide her comments. However, Ms. Kovesdi did not return 
communication until September 15, 2003, only three days before the application would be held 
abandoned by the PTO. See Kovesdi Petition, Exhibit M. Completely disregarding the merits of 
Mr. Rajasekharan's Response and his position of sole inventorship of all but two of the 
dependent claims, Ms. Kovesdi's representatives requested that Mr. Rajasekharan sign off on 
their own prepared response pursuing all claims. See id. Because of the belated nature of this 



5 Mr. Rajasekharan finds it incredulous that Ms. Kovesdi would declare on record in the '952 
Application that "a diligent effort has been made to obtain the signed Declaration of Mr. 
Rajasekharan" when only nine days had lapsed between her making such an assertion and first 
notifying Mr. Rajasekharan of the filing of the '952 Application and requesting that he execute a 
joint declaration for such. In view of the April 4 th Notice to File Missing Parts, this assertion 
was made with over one month remaining before a response was due without extension and over 
six months before any holding of abandonment could ultimately take place. 
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communication, the intermediate four months in which the parties could have addressed the 
merits of any inventorship dispute were squandered by Ms. Kovesdi. 

Mr. Rajasekharan denied to take part in Ms. Kovesdi's proposed Response as indicated in 
the letter of September 16, 2003, which stated that "[w]e will not risk harming Mr. 
Rajasekharan' s interest in the above-reference application by hurriedly executing and submitting 
your 'eleventh hour' response." See Appendix E. Such a denial was also based on the fact that 
Ms. Kovesdi had provided no evidence whatsoever, other than two statements from Mr. 
Rajasekharan in which he believes are taken out of context, to support a determination that she is 
a joint inventor of any claims in addition to the two dependent claims 43 and 44. On September 
22, 2003, Ms. Kovesdi was encouraged to file Mr. Rajasekharan' s response to avoid the 
application being held abandoned, after which the parties could meet "to jointly analyze 
inventorship claim by claim in view of each side's support for conception of the claimed 
invention." See Kovesdi Petition, Exhibit N. In spite of the PTO's strict ruling that 
correspondence to the PTO must be signed by a representative of each party, Ms. Kovesdi 
unilaterally submitted her own prepared Response on September 17, 2003, despite the lack of 
any signature on behalf of Mr. Rajasekharan, and then submitted it again on September 22, 2003, 
with Mr. Rajasekharan' s letter from that day as alleged support for not having his signature. 6 
Not surprisingly, the PTO declined to consider Ms. Kovesdi's two Responses due to the failure 
to obtain Mr. Rajasekharan's signature. See Appendix F. In an Office Action issued on 
December 16, 2003, the PTO gave the parties an additional month to submit a proper response 
signed by both parties. See id. If no response is submitted by June 16, 2004 (the end of the 
response period with extensions), the '952 Application will be held abandoned. 

9 In a communication of April 1, 2004, well into the period requiring extension to respond, 
Ms. Kovesdi's representatives again insisted that Mr. Rajasekharan sign off on their Response 
from September 17, 2003, without even addressing the merits of the inventorship dispute. See 
Kovesdi Petition, Exhibit T. In a communication of April 8, 2004, Mr. Rajasekharan expressed 
his intent not to let the instant application go abandoned and among several options he proposed 
was the filing of a continuation application. See Kovesdi Petition, Exhibit U. 



6 The deadline for responding was extended from the 18 to 22 n due to Hurricane Isabel forcing 
the PTO to close on the 18 th and 19 th . The 20 th and 21 st fell on the following weekend. 
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Rather than taking the straightforward and reasonable procedural step(s) necessary to 
have the PTO ultimately address the claims in the '952 Application in which Ms. Kovesdi 
disputes Mr. Rajasekharan's position of sole inventorship, she chose instead to file a Petition on 
April 19, 2004, requesting in essence that the PTO overturn their Decision of September 17, 
2002, i.e., the PTO grant 37 C.F.R. § 1.47(a) status to Mr. Rajasekharan (even though the PTO 
previously recognized Mr. Rajasekharan's Declaration as compliant with 37 C.F.R. § 1.63) and 
withdraw the requirement of dual representation under MPEP § 402.10 (even though the PTO 
held that such a requirement was deemed necessary). See Kovesdi Petition, Exhibit U. In the 
alternative, Ms. Kovesdi unilaterally requested that the PTO stay the prosecution of the '952 
Application. See id. This extraordinary course of action by Ms. Kovesdi has also been chosen 
over Mr. Rajasekharan's numerous invitations to open a dialog between the parties to discuss 
each side's inventorship contentions. To this day, Ms. Kovesdi simply refuses to have any 
discussions with Mr. Rajasekharan choosing instead to waste both parties' time and money, as 
well as the PTO's resources. 

On May 21, 2004, the PTO dismissed Ms. Kovesdi's April 19 th Petition on the grounds 
that Ms. Kovesdi's arguments "are not persuasive that the relief [sought] can or should, be 
granted." See Appendix G. Particularly, the PTO articulated that "Rule 47 status is not 
applicable to the ['952] application," reiterating that expressed in the PTO's Decision of 
September 17, 2002, and "[t]o grant this petition would essentially render the grant of [Mr. 
Rajasekharan's] petition under 37 C.F.R. § 1.182 of no effect, as petitioner could by such means 
file Office correspondence upon behalf of the recalcitrant inventor, to the prejudice of that 
inventor's rights." Id. Moreover, the PTO suggested that Ms. Kovesdi consider filing a 
continuation or divisional application, the very same course of action that Mr. Rajasekharan 
proposed to Ms. Kovesdi nearly two weeks prior to the filing of her Petition. See id. 

D. The Inventorship Dispute with Ms. Kovesdi 

As noted above, Ms. Kovesdi apparently disputes Mr. Rajasekharan's position that he is a 
sole inventor of sixty eight of the seventy claims found in the '356 Provisional (and hence in Ms. 
Kovesdi's '952 Application). 

In an attempt to resolve the inventorship dispute amicably or at least move the parties 
forward to conduct a substantive dialog on the merits of each side's inventorship contentions 
without PTO intervention, Mr. Rajasekharan at his own initiative provided Ms. Kovesdi with key 
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parts of his supporting documentation along with an explanation for his position of sole 
inventorship for the claims in his instant application on November 13, 2003. See Kovesdi 
Petition, Exhibit P. Unfortunately, Ms. Kovesdi snubbed this offer as well as every other offer 
by Mr. Rajasekharan to open a meaningful dialog. See, e.g., Kovesdi Petition, Exhibit T. 
Although Mr. Rajasekharan' s recognizes that a PTO substantive analysis of the evidence 
pertaining to inventorship is premature at this point as none of the claims in the instant 
application or the '952 Application have been allowed and no interference-in-fact exists, it is 
worthy of note that Ms. Kovesdi has conveniently not disclosed to the PTO her own admissions 
adding support to Mr. Rajasekharan's position. See, e.g., Kovesdi Petition, Exhibit P submitting 
an Email from Ms. Kovesdi to Mr. Rajasekharan (March 14, 2001) ("Your idea is valued at 500 
million dollars!!!"). 

II. MS. KOVESDI LACKS STANDING TO ASK FOR THE RELIEF SOUGHT AS 
SHE IS NOT A PROPER PARTY IN INTEREST TO THE INSTANT 
APPLICATION 

Ms. Kovesdi has no standing whatsoever in the instant application to ask for the relief 
sought as she is not a proper party in interest because she was not included in the inventorship of 
record. As already noted by the PTO in this very application, petitions by those other than a 
proper party in interest will not be treated on the merits. There is no reason to now deviate from 
such a steadfast rule. 

It is well established that prosecution of an application is conducted as an ex parte 
proceeding, i.e., done for the benefit of applicants and/or assignee only ("the Proper Parties in 
Interest"), and without notice to, or argument by, any person adversely interested. See, e.g., 
C.F.R. § 1.33(b) ("Amendments and other papers . . . must be signed by: (1) A registered 
attorney or agent of record appointed . . ., (2) A registered attorney or agent not of record who 
acts in a representative capacity . . ., (3) An assignee . . ., or (4) All of the applicants "). 

The inventorship of the instant application was set forth at the time of filing in a properly 
executed Declaration and Power of Attorney signed by the inventor of record, Mr. Rajasekharan, 
pursuant to 35 U.S.C. § 115-116 and 37 C.F.R. § 1.63. Mr. Rajasekharan appointed, inter alia, 
Rodger Tate and Trevor Coddington of Hunton & Williams (formerly with Brobeck, Phleger & 
Harrison) as his representatives to conduct prosecution of the instant application before the PTO. 
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Ms. Kovesdi is not a proper party in interest, i.e., inventor or assignee, nor is empowered 
to represent the instant application. Accordingly, Ms. Kovesdi lacks standing and the PTO 
should not entertain her present petition by considering its merits. 

III. MS. KOVESDPS PREMISE FOR RECOGNIZING HER AS A PROPER PARTY 
IN INTEREST IS FRIVOLOUS 

Ms. Kovesdi argues that she has a proprietary interest in the matter on the grounds that 

her supplied "evidence demonstrates that Ms. Kovesdi is a co-inventor of the subject matter 

claimed in the '597 application . . ." See Kovesdi Petition, page 8. This argument is premised 

on Ms. Kovesdi' s interpretation of the law as articulated in her Petition as follows: 

Although under MPEP § 2137.01 executors of an oath or declaration under 37 
C.F.R. § 1.63 are presumed to the inventors, it is respectfully submitted that this 
presumption is rebuttable where, as here, the inventorship is in dispute. 
Furthermore, with reference to 37 C.F.R. § 1.47(b) and MPEP 409.03(f), a proper 
party in interest status need not be limited to the named inventors or assignees, 
but also includes one "who otherwise shows sufficient proprietary interest in the 
matter." 

Applicants respectfully submit that this premise and overall argument is unsoundly based and 
frivolous. 

Ms. Kovesdi' s argument is nonsensical and lacks any established legal foundation. No 
statute, PTO regulation, nor precedential decision is cited that would lead one to reasonably 
believe that the PTO's determination of the proper parties in interest is "rebuttable where, as 
here, the inventorship is in dispute" as Ms. Kovesdi argues. Id. Indeed, case law provides that 
the parties executing an oath or declaration under 37 C.F.R. § 1.63 are presumed to be the 
inventors. Driscoll v. Cebalo, 5 USPQ2d 1477, 1481 (Bd. Pat. Inter. 1982). However, Applicant 
can find no case law that suggests that this presumption is rebuttable, particularly in an 
application by someone who is not a party to that application. Applicants respectfully believe 
that no legal precedent exists that would enable the PTO to recognize Ms. Kovesdi as a proper 
party in the instant application after the inventive entity has already been designated by 
Applicants under 37 C.F.R. § 1.63. Moreover, Ms. Kovesdi's reliance on 37 C.F.R. § 1.47(b) 
and MPEP § 409.03(f) is misplaced. These sections apply to inventors refusing to sign or who 
can not be reached. This application clearly has no § 1.47 status as Mr. Rajasekharan has at the 
time of filing submitted an executed sole Declaration in accordance with 37 C.F.R. § 1.63. Any 



Page 9 of 13 



Application No. 09/987,597 

Reply to Kovesdi Petition of May 24, 2004 

attempt by Ms. Kovesdi to prove that she has a "sufficient proprietary interest in the matter" 
under 37 C.F.R. § 1.47(b) is simply irrelevant. 

IV. MS. KOVESDI HAS FAILED TO SHOW GOOD CAUSE FOR THE PTO 

GRANTING HER EXTRAORDINARY AND INAPPROPRIATE REQUEST 

The relief sought by Ms. Kovesdi is predicated on the assumption that the PTO's 
decisions in this application and the '952 Application "have practically precluded consideration 
of the inventorship issue on the merits and have given Mr. Rajasekharan control over both the 
'952 and the '597 applications." See Kovesdi Petition, page 2. This assumption that Ms. 
Kovesdi relies upon is unsoundly based and therefore, Ms. Kovesdi has failed to show good 
cause for the PTO granting her extraordinary and inappropriate request. 

Ms. Kovesdi finds herself "practically precluded" only because she and her counsel fail 
to recognize and take the necessary and reasonable step(s) that would lead to the PTO possibly 
considering the inventorship dispute. Any contention that Mr. Rajasekharan or the PTO has 
encumbered her ability to ultimately have an interference declared is unfounded. The only 
position taken by Mr. Rajasekharan that seemingly stifles Ms. Kovesdi 's prosecution tactics and 
objectives is his concurrence with her that she is a joint inventor of two, and only two, dependent 
claims in the '952 Application. Although Mr. Rajasekharan seeks an ultimate resolution to the 
inventorship dispute as well, he nor his counsel are believed to be under any obligation that 
would require them to specifically point out to Ms. Kovesdi and her counsel the straightforward 
prosecution steps that can and should be taken by her to bifurcate the claims not in dispute from 
those that could ultimately wind up in an interference proceeding. Even the PTO has suggested 
that Ms. Kovesdi should consider filing a continuation or divisional application off of the '952 
Application in order to prosecute further without needing any cooperation of Mr. Rajasekharan. 
See Appendix G. This is the very same course of action that Mr. Rajasekharan proposed to Ms. 
Kovesdi nearly two weeks prior to the filing of her Petition in the '952 Application. 
Accordingly, there is no need for the PTO to step in to save Ms. Kovesdi when other reasonable 
options exist. Any loss of rights by or irreparable harm to Ms. Kovesdi will be at her own fault, 
not any fault of Mr. Rajasekharan or the PTO. Ms. Kovesdi' s present request not only wastes 
her own time and resources, but that of Mr. Rajasekharan as well as the PTO. 

Mr. Rajasekharan has every right to control the prosecution of the '597 Application and 
has no more control over the '952 Application than Ms. Kovesdi. Mr. Rajasekharan is 
designated as the sole inventor in the '597 Application and therefore can conduct ex parte 
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prosecution without notice to, or argument by, any person adversely interested, Le. 9 Ms. Kovesdi. 
See Remarks II, supra. In the '952 Application, Mr. Rajasekharan simply refuses to be bullied 
by Ms. Kovesdi's all or nothing tactics, i.e., her insistence on filing nothing, but her own 
prepared response, particularly in view of her unsupported dismissal of Mr. Rajasekharan's 
belief that sixty-eight of the seventy pending claims are solely conceived by him. Any 
suggestion that the current balance of power weighs in Mr. Rajasekharan's favor is erroneously 
based. Any veto power is equally shared among the Applicants in the '952 Application. For 
example, Ms. Kovesdi has wielded the very same veto power by refusing to submit or even 
consider Mr. Rajasekharan's prepared Response. 

Ms. Kovesdi's further assertion that Mr. Rajasekharan has acted improperly because he 
"has not provided, either to the Patent Office or to Ms. Kovesdi, any evidence of his asserted sole 
conception of the claims in either the '597 application or with respect to all but two claims in the 
'952 application" is senseless. Mr. Rajasekharan has not provided the actual support for his 
claim of inventorship to the PTO because he and his undersigned counsel are well aware that the 
there is no duty to do so and more importantly, the PTO prefers not to get involved in 
disagreements among the inventors until allowable subject matter exists and an interference can 
be provoked. No allowable subject matter exists so far in either the '952 or '597 Applications 
and therefore an interference is premature at this point in time. If the PTO wishes to presently 
consider the merits of each side's contentions, Mr. Rajasekharan is more than happy to provide 
his evidentiary documentation supporting a position of sole inventorship. Contrary to Ms. 
Kovesdi's assertion, Mr. Rajasekharan has invited Ms. Kovesdi to participate in a meaningful 
dialog to address the merits of each side's inventorship contentions as a means to reach to a 
jointly submitted Response suitable to both parties without the PTO's involvement. For 
example, on November 13, 2003, Mr. Rajasekharan not only provided Ms. Kovesdi with the key 
parts of his supporting documentation, but also an explanation for his position of sole 
inventorship. See Kovesdi Petition, Exhibit P. Instead, Ms. Kovesdi has ignored all invitations 
to discuss inventorship preferring to improperly and/or prematurely to use prosecution before the 
PTO as an inventorship dispute resolution forum. 
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V. THE DISPUTED CLAIM OF JOINT INVENTORSHIP IS NOT RIPE FOR THE 
PTO'S CONSIDERATION 

In essence, Ms. Kovesdi seeks to have the merits of each side's contentions for 
inventorship immediately judged by the PTO. Applicants respectfully submit that Ms. Kovesdi's 
disputed claim of joint inventorship is not ripe for the PTO's consideration as no interference-in- 
fact exists. 

Although interferences typically involve date of invention contests, an interference can be 
declared to establish inventorship. See Chou v. University of Chicago, 254 F.3d 1347, 1358 n. 2, 
59 USPQ2d 1257, 1262 n.2 (Fed. Cir. 2001) (a means for a putative inventor to assert 
inventorship rights is to file a patent application and seek to have the PTO declare an interference 
in order to establish inventorship); Linkow v. Linkow and Edelman, 517 F.2d 1165, 186 USPQ 
223 (CCPA 1975). An interference between pending applications may be requested by an 
applicant who has become aware of another application which may be claiming the same 
invention. MPEP § 2300.02. Before an interference is declared between two or more 
applications, the examiner must be of the opinion that there is interfering subject matter claimed 
in the applications which is patentable to each applicant subject to a judgment in the interference. 
37 C.F.R. § 1.603. The parties have a dispute only because the 'claims' in their respective cases, 
interfere with each other. Louis v. Okada, 57 USPQ2d 1430, 1431-34 (BPAI 2000). If no 
'claim' of one party interferes with at least one 'claim' of another party then there can be no 
interference-in-fact. Id. interpreting 37 C.F.R. § 1.601 (j). 

Consideration of Ms. Kovesdi joint inventorship contentions is presently premature. Ms. 
Kovesdi does not have a patent application that would stand in the way of examining or issuing 
the instant application. No claim in the '952 Application or the instant application has been 
found allowable by the Examiner yet. Because Ms. Kovesdi has no claims that actually 
"interfere" with those in the instant application, there is no inventorship dispute ripe enough for 
the PTO to adjudicate at this point. 
VL CONCLUSION 

The present Kovesdi Petition is another example of an extraordinary attempts to 
contravene and circumvent well established PTO rules. Ms. Kovesdi has a long history of filing 
frivolous Petitions as evidenced by the PTO's previous refusal to allow her to join in the 
prosecution of the instant application and the dismissal of her recent petition in the '952 Joint 
Application. The present Petition is nonsensical in view of the well established principles of ex 
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parte prosecution and her failure to implement the reasonable steps needed to seek an 
interference with the presently recited claims. In essence, because Ms. Kovesdi and her counsel 
do not comprehend how to proceed under the PTO's rules, they ask that the PTO change the 
rules to accommodate them at the other party's expense. Applicants urge the Commissioner to 
deny the relief sought by Ms. Kovesdi. 

No fee is believed to be necessary for this submission. However, in the event that the 
PTO deems a fee is necessary for consideration of the remarks herein, please charge such fee to 
the undersigned's Deposit Account No. 50-0206. 

Respectfully submitted, 




Dated: June 14, 2004 By: 

Rodger L. Tate 

Hunton & Williams LLP Registration No. 27,399 

Intellectual Property Department 

1900 K Street, N.W., Suite 1200 

Washington, DC 20006-1109 

(202) 955-1500 (telephone) 

(202) 778-2201 (facsimile) 
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System and method for authoring and providing information relevant to a— 
phyninalthe 



Abstract 

A system and method capable of reading machine -readable labels from physical 
objects, reading coordinate labels of geographical locations, reading 
timestamp 

labels from an internal clock, accepting digital text string labels as input 
obtained directly from a keyboard type input device, or indirectly using a 
speech- to- text ^ncring transforming anv other label type information encoding 
into digital data bv some transduction means, and treating these different 
labels uniformly as object identifiers for performing various indexing 
operations such as content authoring, playback, annotation and feedback. The 
system further allows for the aggregating of object identifiers and their 
associated content into a single addressable unit called a tour. The system 
can 

function in an authoring and a playback mode. The authoring mode permits new 
audio/text /graphics/video messages to be recorded and bound to an object 
identifier. The playback mode triggers playback of the recorded messages when 
the object identifier accessed. In the authoring mode, the system supports 
content authoring that can be done coincident with object identifier creation 
thereby enabling authored content to be unambiguously bound to the object 
identifier. In the playback mode, the system can be programmed to 
accept/solicit 

annotations/ feedback from a user which may also be recorded and unambiguously 
bound to the object identifier. 

Claims 

What io claimed io! 
I claim: 

1. A method for authoring information relevant to a physical world, 
comprising: 

detecting with an authoring device a first label associated with a first 
object; 

and triggering, in response to detecting, a system for authoring content; 
wherein the content is to be unambiguously bound to the first object and is 
to 

be rendered on a playback device during detection of the first label. 

2. The method as recited in claim 1, wherein the system for authoring content 
is 

resident on the authoring device. 

3. The method as recited in claim 1, wherein the authoring device and the 
playback device are integrated within a single apparatus. 

4. The method as recited in claim 1, wherein the label is selected from a 
group 

consisting of a barcode label, a coordinate, a RFID tag, an IR tag, a time 
stamp, a text string, and n npnonh t-n f.mch ntrina.any other label type whose 
information can be 

transformed to d igital data by some transduction means. 



5. The method as recited in claim 1, wherein the content is selected from a 
group consisting of audio, text, imacrGgraphics . and vidgo r or a combination 
thereof . 

6. The method as recited in claim 1, wherein the content is a link to a live 
agent . 

7. The method as recited in claim 1, further comprising the steps of 
detecting a 

second label associated with a second object; triggering, in response to 
detecting, the system for authoring content which is unambiguously bound to 
the 

second object; and aggregating the content bound to the first object and the 
second object into a single l ogical entity called a tour. 

8. The method as recited in claim 1, further comprising the step of detecting 
a 

second label associated with the first object and normalizing the first label 
and the second label such that the content bound to the first object can 
rendered during detection of either the first or second label in the playback 
mode . 

9. The method as recited in claim 1, further comprising the step of storing 
the 

content in non-volatile memory resident in the apparatus. 

10. The method as recited in claim 1, further comprising the step of 
uploading 

the content to a remote server. 

11. The method as recited in claim 10, wherein the step of uploading is 
performed via a wireless network. 

12. The method as recited in claim 10, wherein the step of uploading is 
performed via a wired network. 

13. A computer -readable media having instructions for authoring information 
relevant to a physical world, the instructions performing steps comprising: 
detecting a first label associated with a first object; and triggering, in 
response to detecting, a system for authoring content to be unambiguously 
bound 

to the first object; wherein the content is to be rendered during detection 
of 

the first label by a device in a playback mode. 

14. The computer-readable media as recited in claim 13, wherein the 
instructions 

perform the further steps of detecting a second label associated with a 
second 

object; triggering, in response to detecting, a system for authoring content 
to 

be unambiguously bound to the second object; and aggregating the content 
bound 

to the first object and the second object into a single loq|cal entity called 
tour. 



15. The computer- readable media as recited in claim 14, wherein the 
instructions 

perform the further step of detecting a second label associated with the 
first 

object and normalizing the first label and the second label such that the 
content can rendered during detection of either the first or second label by 
the 

device in the playback mode. 

16. A computer-readable media having instructions for authoring content to be 
associated with objects in a physical world, the instructions performing 
steps 

comprising: normalizing a read object label associated with an object into an 
object identifier; placing the object identifier into a databaoc - jLiuteft tftfrlQ 
repository : 

accepting content to be rendered when the object label is read in a playback 
mode; and binding the content to the object identifier in the databaoc . jin^ex 
table 

17. The computer- readable media as recited in claim 16, wherein the 
instructions 

allow a plurality of different label types to be normalized to one object 
identifier. 

18. A method for providing information relevant to a physical world, 
comprising: 

detecting with a device a label associated with an object; normalizing 
information contained in the detected label into an object identifier; using 
the 

object identifier to search n rintnhnaoan index table repository to find 
content bound to tho object 

ftftft pb-ieet identifier: and rendering the content. 

19. The method as recited in claim 18, further comprising the step of 
retrieving 

the content bound to the object identifier from local memory in the 
apparatus . 

20. The method as recited in claim 18, further comprising the step of 
retrieving 

the content bound to the object identifier from a remote server. 

21. The method as recited in claim 18, wherein the content is selected from a 
group consisting of audio, text, im^araphics. and video. 

23.22. The method as recited in claim 18, wherein the label is selected from 
a 

group consisting of a barcode, a coordinate, an IR tag, a RFID tag, a 
times tamp, 

a text string, and n npnnnhany other label type whose information can be 

trana formed to text otring . 

di gital data by some transduction means. 

24. 23. The method as recited in claim 18, wherein the content is a connection 
to a 

live agent. 



25t 2 4. The method as recited in claim 18, further comprising the step of 
determining the current time and comparing the current time to the timestamp 
before rendering the content. 

g^25. The method as recited in claim 18, wherein the step of rendering the 
content 

comprises streaming the content from a remote server. 

^^26. The method as recited in claim 18, further comprising the steps of 

accepting . . , 

annotations/feedback after the rendering of the content and binding the 
annotations /feedback to the object identifier. 

^27. The method as recited in claim WriL further comprising the step of 
storing 

the annotations/feedback in local memory. 

^28. The method as recited in claim ZZ-tZLl. further comprising the step of 
storing 

the annotations/ feedback in a remote memory. 

a^29. A computer- readable media having instructions for providing 

information , . . 

relevant to a physical world, the instructions performing steps comprising: 
detecting a label associated with an object; normalizing information 

contained , , » » ^ . , _ . ^ . _ 

in the detected label into an object identifier; using the object identifier 

search - ***** repository to find content bound to the 

object identifier; 

and rendering the content. 

31. 30. T he computer- readable media as recited in claim ^rZS^ wherein the 

content is . 
selected from a group consisting of audio, text, graphics, and video. 

32, 31. A method for providing information relevant to a physical world, 
comprising: 

storing an object identifier indicative of a plurality of read labels 

associated . 

with an object into - *^^nn* n ****** table repository ; and using the 

ri-.j-nln-inn -in HAy table _ 
r^poflitorv to bind content to the object identifier and, accordingly, the 
object; whereby the content is renderable when any one of the plurality of 
labels is detected in a playback mode. 

^32. The method as recited in claim ^hr,3JU wherein at least one of the 
plurality of 

labels is mnl-^m nmnfo^ lrftadY pr es ent on t he object . 

^33. The method as recited in claim &r22*. further comprising the step of 
attaching 

at least one of the plurality of labels to the object. 

^34- The method as recited in claim Z*r32u- wherein the plurality of labels 



is 



selected from a group consisting of a barcode label, a coordinate, a RFID 
tag, 

an IR tag, a time stamp, and a text strings, Qy any Qtfrer lafrsl type wftoss 
^fnrmation that C an bp t-.ranfff ormed to digital data by gQmft transduction , 
means ■ 

.3^35. The method as recited in claim ^3-^31. further comprising the steps of 
detecting 

the plurality of labels. 

37. 36. A method for providing information relevant to a physical world, 
comprising: 

associating one or more labels with each of a plurality of objects in a tour; 
storing an object identifier indicative or the one or more labels associated 
with each of the plurality of object in the tour in a databaooan Anfley table 
repository ; 

authoring content relevant to each of the plurality of objects in the tour; 
and 

binding the content to an object identifier in the databaoojfldex table 
rfiPfffljfrPffY which 

corresponds to the relevant one of the plurality of ob}ects xn the tour 
whereby 

the content is renderable when the label is detected by a playback device 
without regard to the order in which the content was authored. 

38. 37. The method as recited in claim 4^-36. wherein the labels are selected 
from a 

group consisting of coordinates, barcode labels, RFID tags, IR tags, 
times tamps, 
and text. 

t«efc str i ng, and anv label tvne whose information that can bfl transformed tp 

digital data bv pome transduction means. 

3Q. 38. A system for authoring and retrieving selected digital multimedia 
information relevant to a physical world, comprising: a plurality of machine 
readable labels relevant to the physical world; an apparatus for detecting 
the 

machine readable labels and including programming for normalizing information 
contained in the detected label into an object identifier; and a digital 
multimedia nvim^nntimfc collection accessible by the apparatus storing 
content 

indexed by the object identifiers. 

a « r-ecited in olaim 38. wherein the apparatus further 

- T - -ithor i n^ digital multimedia in response to detecting one of the 

pluralit y of labels whieh ia to be stored within the digital multimedia 

col^ec^ion a pfl imamhigiiQualy bound to the object identifier, 

40. The system as recited in claim 39, wherein the apparatus further 
comprises a 

oyotcm for authoring digital multimedia in rooponoc to detecting one of the 
plurality of labclo which io to be otorod within the digital multimedia 
library 

and unambiguouoly bound to the object identifier. 




4 1. The o y uL cm n n i^'i. 

eemfneieee-*- ... milltime dia in response to detecting one of the 

^eTfoTrendering dxgxtal .™lt^* * rendered being the content 
Surality of labels the dx mult^d associated wifch a detecte d 
Unambiguously bound to the object xa 

t as recited in claim ^ wherein the digital multimedia 
43_4J^ The system as recitea xi 

*S23S£55Si-- o M or «. o £ audio — - - 



capabilities 



, . - recited in cl.i- the 

44 The system as recitea 

wriee e £o Jtt ^d^^g^^^ efeiie ^^ 



• i - m «_45^ wherein the d x y i L al H-imnrti n - 

46 The system as recited xn claxm 

±*tea*y^siisl^^ 



47 The system as recitea 
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via rni ■ 

n m a*-38 wherein the w xu l u u nr r w nr k - 

49. The system as recited xn claxm £r£U 

ocllular tnlrepheRe iictworlr 

^^^^^ . n . __3- wh erein the dig^aWiwdia- 

= = rprited in claxm *9rjisj= WI1B1C 

50. The system as recitea xn 

51. The system as retn.cu 



digital multimedia library via the Internet. 

52. The oyotom ao recited in claim 30, wherein the apparatuo aeooooco the 
digital multimedia library via a voice portal. 

53. The oyotcm ao recited in claim 30, wherein the apparatuo acccooco the 
digital multimedia library via a cellular telephone voice mailbox. 54. The 
oyotcm ao recited in claim 39,3 JL. wherein the digital multimedia is 
aggregated into a tour. 

j ^52. The system as recited in claim 3 ^38. wherein the digital multimedia 
is 

randomly accessible by the apparatus. 

Ji6 ^53. The system as recited in claim ^9 ^38 , wherein the digital multimedia 
is 

accessible by the apparatus in a sequential order. 

The system as recited in claim 3^ -38. wherein the apparatus comprises 

a 

personal digital assistant . 

SS-rgSu. The system as recited in claim 3^ -38. wherein the apparatus comprises 
a 

cellular telephone. 

The system as recited in claim 3^ -38. wherein the apparatus comprises 

purpose 

built devices targeted to a specific application. 

6^ -57 . An apparatus for authoring information relevant to a physical world, 
comprising: circuitry for detecting a label associated with an object; and a 
system for authoring content to be unambiguously bound to the object as 
represented by the detected label which content is to be rendered during 
detection of the label in a playback mode. 

6- W58. The apparatus as recited in claim ££-^57. wherein the circuitry 
comprises a 
barcode reader. 

6-3-^59. The apparatus as recited in claim 6^-57. wherein the circuitry 
comprises an IR 
tag reader. 

The apparatus as recited in claim 6S^57 . wherein the circuitry 
comprises a RFID 
tag reader. 

64^ -61. The apparatus as recited in claim 6-6- ^57. wherein the circuitry 
comprises a 

keyboard for inputting textual information. 

62. The apparatus as recited in claim 57. wherein the circuitry comprises of 
analog to digital inf ormation transducer. 

65- ^63 . An apparatus for authoring and providing information relevant to a 
physical 



. . -^^t-rv for detect inq a label associated with an object; 

IZ 1 contained in the detected label 

afobject identifier; a system for authoring content in an authoring mode 
content is to be unambiguously bound to the object identifier; and a system 

rendering content in a playback mode, the content ^^f^^S"^ 
unambiguously bound to the object identifier assocxated with a detected 

label . 

The apparatus as recited in claim S^SL, further comprising a 

nrror^ownLading authored content to a remote location and for retrieving 
content from the remote location for rendering. 

Wrfa* The apparatus as recited in claim «rfa-. further comprising a memory 



for 

storing the content. 

^£L. The apparatus as recited in claim wherein the circuitry 

comprises a 
barcode reader. 

&^£U The apparatus as recited in claim e*r£L. wherein the circuitry 
comprises an IR 
tag reader. 

The apparatus as recited in claim wherein the circuitry 

determines a 
coordinate location. 

The apparatus as recited in claim *±t&2U wherein the circuitry is a 

RFID tag 
reader. 

7 a a ,v^ r rir - in nm« <n. mmm-ln tlm flimiltTY 1 n m analog 

fco 

Description 

CnOCC REFERENCE TO R ELATED APPLICATION 

[0001] THi i .i- r 1irn-- ^ r — '"^ tion claims the benefit of priority to 

ApplicSof ZZJ^eo,********** filed on 0U1. 18. 
incorporatca\ 2fiJLL». 

herein by reference in ito entirety. 
BACKGROUND OF THE INVENTION 
[0002] 1 - Field nf Tnvention 

r Q Q031 This invention relates generally to information systems and, me*e- 



particularly, rclatoo to a system and method for authoring and providing 
information 

relevant to a physical world. 

rq Q041 2. Descriptio n " f hhe Related Art 

f-Q-Q-Q ^QQQS I The exponential growth of the Internet has been driven by three 

factors, , 
namely, the ability to author content easily for this new medium, the simple 
^^-c^^n-i. e_ q .. urUfnrm-r^oyr^ locator f "URL" ) . based indexing scheme 

for , 
content organization, and the ease of accessing authored content^ 4e.g., by 

just 

a mouse click on a hyperlink^. However, attempts made to emulate the success 

the Internet in the mobile device usage space have not been very successful 

to , 
date. The mobile device usage space is the whole physical world we live in 

and ' ^ V. 

unlike the tethered ^rftrfmT^ 1 «7™"»" fc ' Mf f,,pc,,> based Internet world where 
all 

objects are virtual, the physical world is composed of real objects, 
geographical locations, and temporal events.*. 4which occur in isolation or in 
conjunction with an object or location^-. These diversities pose problems not 
present in the existing Internet world where all virtual objects can be 
uniformly addressed by a URL. Thuo, there cxioto a nood 

for a ocheme that addrcooco the labeling of objecto, locationo and temporal 
evento, a genome that hao an indexing method which trcato thcoe different 
labelo 

uniformly and tranoparcntly to the underlying labeling method, a ocheme that 



help author content ocamloooly for thcoc different phyoical world cntitioo 
and 

bind the content to the indicco, and a ocheme that can provide caoy aoccoo 

and . , . 

playback of the authored content for any real world entity, e.g., object, 

location and temporal evento. 

r 4U ^0006 1 Attempts have been made to build applications that enable seamless 
browsing of just one domain, such as the domain of physical objects or the 
domain of geographical locations. There have also been attempts to treat 
browsing of objects and locations together. However, these attempts fail to 
address the key factors mentioned above that made the Internet what it is 
today, 

i.e., the most effective medium for information dissemination. In particular, 
these attempts do not Affectively address the labeling issue, whichi.e. , 

i! |fQmation ? Qf different fo rmats aci-Qflfl different labeling schemes. This is a 

problem unique to the physical world and not present in the PC-based virtual 
browsing method 4-where all content in the virtual world can be addressed by a 
URL ) , they do not have a uniform indexing ocheme acrooo different labeling ■ 
.nhnmnn. Moreover , they do not support authoring of content that is bound to 
these 

different label types, they do not oupport content authoring on the device 
(which is a key 

deficiency given that on-device content authoring is the most natural, 



efficient, and error-free method for most mobile device usage scenarios), and- 
they do not oupport a QX^ 

playback of content indexed by the different labeling schemes. 

ra^a^ -OPCm To enable seamless mobile browsing which envelops all of these 
apparently 

disparate application domains these deficiencies need to be addressed. The 
absence of a labeling and content binding scheme makes it very hard for one 
to 

do custom labeling of objects and bind content to the labels (the oolution 
of fcrod by proocntly known oyotomo would bo a manual error prone proccoo) . 
The absence of an 

annotation/feedback binding scheme makes it very hard to maintain the 
correspondence between the content and the annotation/f eedback. The absence 
of 

seamless bridging of location-based, object-based, events-based, ajs^ 
conventional web hyperlink based services requires different 
devices/applications to navigate these different domains. 

[ nnnr] PurmnHy f>inrnOQQ81 There are four separate application domains in 
the mobile device space, 

namely, object-based devices and applications, coordinate-based devices and 
applications, *- -j mnnhnmp^TT^^al based devices and applications, and 
traditional URL-based 

devices and applications. Object-based devices can read labels off of 
physical 

objects fn a. v ia harrnHp.g^aA-. radio- frequency identification ( "RFID-H*f*4-") , 

lIlR"f r taasf^_ and are typically used in a proactive fashion where a user 
scans 

the object of interest using the devices. These devices attempt to support 
browsing the world of physical objects in a manner that is similar to surfing 
the Internet using a web browser. The coordinate-based application domain is 
an 

emerging domain capitalizing on the knowledge of geographical 

InnnM nr. locations made 

available through a variety of location detection schemes minh nn GPSbased on 

ginhal-posifcioning sy stem ("GPS"), an aBSisted-GPS ( »ft -GPS-r-^Q ft" ) where 
satellite 

signals mav be weak, an angle of arrival ( "AOA" ) system, or a time difference 

arrival ( " TDOA" ) *q4= ^sysfcem . An existing application domain in the PC-world, 
e.g. , 

timeline based information presentation, is also making inroads into the 
mobile 

device space. However, no devices or applications presently exist that are 
capable of bridging these different application domains in a near seamless 
and 

transparent manner. 

rfrQ-a-g -0009 1 In the field of portable interactive digital information systems 
that 

employ device -readable object or location identifiers several systems are 
known . 

For example, U.S. Pat. No. 6,122,520 describes a location information system 



which uses a positioning system, such as the Navstar Global positioning 

system, , . 

in combination with a distributed network. The system receives a coordinate 
entry from the GPS device and the coordinate is transmitted to the 
distributed 

network for retrieval of the corresponding location specific information. 
Barcodes, labels, infrared beacons and other labeling systems may also be 
used 

in addition to the GPS system to supply location identification information. 
This system does not, however, address key issues characteristic of the 
physical 

world such as custom labeling, label type normalization, and uniform label 
indexing. Furthermore, this system does not contemplate a tour like 
paradigm-^ 

i.e., a "tour" as media content grouped into a logical aggregate. 

teAO frOOlO l U.S. Pat. No. 5,938,721 describes a task description database 
access ible 

to a mobile computer system where the tasks are indexed by a location 
coordinate. This system has a notion of coordinate-based labeling, 
coordinate-based content authoring, and coordinate triggered content 

playback. . 
The drawback of the system is that it imposes constraints on the capabilities 

of 

the device used to playback the content. Accordingly, the system is deficient 

that it fails to permit content to be authored and bound to multiple label 
types 

or support the notion of a tour. 

U^fts ^ooin U.S. Pat. No. 6,169,498 describes a system where location-specific 
messages are stored in a portable device. Each message has a corresponding 
device -readable identifier at a particular geographic location inside a 
facility. The advantage of this system is that the user gets random access to 
location specific information. The disadvantage of the system is that it does 
not provide information in greater granularity about individual objects at a 
location. The smallest unit is a ^site^ (a specific area of a facility) . 
Another 

disadvantage of the system is that the user of the portable device is passive 
and can only select among pre-existing identifier codes and messages. The 
user 

cannot actively create identifiers nor can he/she create or annotate 

associated t 

messages. The system also fails to address the need for organizing objects 

into 

meaningful collections. Yet another disadvantage is that the system is 
targeted 

for use within indoor facilities and does not address outdoor locations. 

f ^ ^0012 1 U.S. Pat. No. 5,796,351 describes a system for providing 
information 

about exhibition objects. The system employs wireless terminals that read 
identification codes from target exhibition objects. The identification codes 
are used, in turn, to search information about the object in a data base 
system. 

The information on the object is displayed on a portable wireless terminal to 
the user. Although the described system does use unique identification code 



assigned to objects and a wireless local area network, the resulting system 
is a 

closed system: all devices, objects, portable terminals, host computers, and 
the 

information content are controlled by the facility and operational only 
inside 

the boundaries of the facility. 

f^4-4 -0Q131 U.S. Pat. No. 6,089,943 describes a soft toy carrying a barcode 
scanner 

for scanning a number of barcodes each individually associated with a visual 
message in a book. A decoder and audio apparatus in the toy generate an audio 
message corresponding to the visual message in the book associated with the 
scanned barcode. One of the biggest drawbacks of this system is the inability 
to 

author content on the apparatus itself. This makes it cumbersome for one who 
creates content to author it for the apparatus, i.e., one has to resort to a 
separate means for authoring content. It also makes it harder to maintain and 
keep track of the association with the authored content, object identif iers^, 
and 

the physical object. 

f 0012 0014 1 U.S. Pat. No. 5,480,306 describes a language learning apparatus 
and 

method utilizing an o ptical identifier as an input medium. The system 
requires 

an off-the-shelf scanner to be used in conjunction with an optical code 
interpreter and playback apparatus. It also requires one to choose a specific 
barcode and define an assignment between words and sentences to individual 
values of the chosen code. The disadvantages of this system are the 
requirement 

for two separate apparatus making it quite unwieldy for several usage 
scenarios 

and the cumbersome assignment that needs to be done between digital codes and 
alphabets and words. 

r&ft ^oo^s l U.S. Pat. No. 5,314,336 describes a toy and method providing audio 
output 

representative of a message optically sensed by the toy. This apparatus 
suffers 

from the same drawbacks as some of the above-noted patents, in particular, 
the 

content authoring deficiency. 

f -ft&4r 40016 1 U.S. Pat. No. 4,375,058 describes aaa apparatus for reading a 
printed code 

and for converting this code into an audio signal. The key drawback of this 
system is that it does not support playback of recorded audio. It also 
suffers 

from the same drawbacks as some of the above-noted patents. 

f 3^-5 -0017 1 U.S. Pat. No. 6,091,816 describes a method and apparatus for 
indicating 

the time and location at which audio signals are received by a user-carried 
audio-only recording apparatus by using GPS to determine the position at 
which a 



particular recording is made. The intent of this system is to use the 

purees a means to know where the recording was done as opposed to using 

binding for subsequent playback on the apparatus or for feedback or 

^ding^Also, the timestamp usage in the system fails to contemplate using a 
timesSmp Is a trigger for playback of special temporal events or binding a 
timestamp to objects, coordinates^ and labels. 

[W±£fiai&] m addition to the patents listed above, which arq all 
^y?!^ ^ rT1 T^tY t ^re are^«*e*e*e other systems on 

wnosfcommon objective is to link printed physical world i-Jf^JJ;" * 
virtual internet URL. More specifically, these systems encode URLs into 
proletary barcodes. The user scans the barcode in a catalog and her web 
browser i7launched to the given URL. Di uum lm of c n ^ai i x uJ who unr t hi n 
approaoh au AirCH n (http i //www.alralio . com) , CoCodo 

fU Ltu S.'vtov jn—^" ■■'■"^ -mri niaitali Convergence 

LLp !// 2 S^rllZ un . Li- j cnn n n o m ^he advantage of these systems is that 

they link the physical world to the rich information source of the Internet. 

disadvantages of these systems are that the URL is directly encoded in the 
barcode and cannot be modified and there is a on e-to-one_mapp ing between a 
physical object and digital URL information. D-i.roxnt. Tnr 

(ULL p ./.W y nm--^ ' r m,Hrtnn nfonni another conYftntiona l system-*fea* 

uses standard iir^ arflal nrc-dnct code CUPCJLL T .. prnet The 

barcode scanning for product lookup and price comparison on the Int ernet . The 
advantage of L hc Darroint£ bia system is that it does not require a 

Sevfce'and^rerris an indirection when mapping code to information instead 

hard-coded, direct URL links. Nevertheless, all of the above systems 
disadvantageously treat each object, i.e., each barcode, as an individual 

a^do not provide a means to create logical relationships among the 

^physical objects at the same location. Another disadvantage of these 

iftha? they do not enable the user to create a personalized version of the 
information or to give feedback. 

SUMMARY OF THE INVENTION 

rnnoni -iw— ~ ■ ■ for a scheme thnf *<1dr*» B e S ttift labeling 

ZxlVjj Tli^ HHP author nrni^t seamiest 

^ f f„r — ph yni — ~" -MM*- »nd Hi« contwil- to thft indices, 

t-.ha, r-m rr-ir^- "°° T — ™* m^rlr of rh* mifhorpd content 

for 



lt 4 tYi ft ii ft- - a p * 



jn ^ro02n To address febe£bia Heedeflgfld, and overcome the deficiencies 

described abeveia-J&S— . , . , . 

^ at-ad art , the f>*eeeftteiasESa£iXft invent ions aa£gB& is embodied in a oyotem 

aaeV-method for authoring and „ m ^ j a „ 

providing information relevant to a physical world. Generally, «*eflna_§&- 
annaratua and system utilinco a ■ ^ , , _ 

"mol nvino y^h a met-^H . Prafprahlv. a h and-held device i-hat is capable of 
reading one or more labels such as, for mreunplohit, iwt Mipit^d tQ . a barcode, 

IR R feeLeSia, location coordinates, and a-timestamp, and for authoring and 

media n content relevant to the labels^B-fehe- iff utilised., IB tftff authoring 

repressing objects, locations, temporal events, SasL-text strings^-efee^ are 
identified and translated into object identifiers which are then bound to 

cogent that the author records for that object identifier. Media content can 

grouped into a logical aggregate called a tour. A tour can be thought of as 

aggregation of multimedia digital content, indexed by object identifiers. **- 

pta^SJcf^de, the authored content is played when one of the above mentioned 
labels (barcode, RFID tag, location coordinates, etc.) is read and whose 
generated object identifier matches one of the identifiers stored earlier in 

tour. The system also enables audio/ J _text/ J _graphics/ J _aad»video annotation 

recorded and bound to the accessed object identifier. Binding to the accessed 
object identifier is also done for any audio/^text^_graphicsf u _ a r_video 
feedback 

provided by the user on the object. 

[00 10] il b ette r un iir rnfi"^-' J nf ^ nhjnni-n 00321 The foregoing i and Otihfir 
and advantages , features, properties and rclationohip n of the 

'! V !S tfa^rent from the following detailed , more particular d escription 

^■^f fha preferred c „ v 

,w^ m «nt-. g ^ t-t,« ■invep t-ion. the accompanying drawings which act forth 

illuotrativc cmbodimcnto, and the claims. _ 

and which a rc indicative of the variouo wayo in which the j ji mciplcc of t ho 

invention may be employed. 

BRIEF DESCRIPTION OF THE DRAWINGS 

\m^QQ2X\ For a ^^"re complete understanding of the pxjyuia^invention^ 



an A advantage t-Waof . reference may be had in now made to preferred 

cmbodimcnto ohown in -the following dencriPtionB 

«-alr OT in ^-"'Hnn wit-h hhp accompanying drawings in which: 



[0 Q20 QQ241 FIG. 1 illustrates an embodiment of the preocnt invention in the 

fl v fl hAm need , fftr four authoring, storage, retrieval, 
of a tour of a ohopping center; 
and playfracH; 

rn05100251 FIG. 2 illustrates ^ MnnV Hi irjir-iTn^pp ^i, flat ion domains of afir- 
Q3ccmplary computer network various Ifrbel tYP6g ag a ■ 
architOGturc for oupporting tour applicationo; 

tyn ^i^n o f fh ft S j 2e P f ^ Qbiect being labeled and the detection range p£ 

the 
1 frft ft 1 ■ 

[W*fifi2i] FIG - 3a illustrates an exemplary tree structure for an instance of 
a 

tour ; 

toa ^oo27 l FIG. 3b illustrates exemplary file formats supported by a tour; 

rAA^ -0028 1 FIG. 4 illustrates examples of bindings that may occur during the 
labeling, authoring, playback, annotation^ and feedback stages of a tour; 

rAO^ .0029 ] FIG. 5a illustrates various label input schemes, label encoding, 
and label 

normalization process and their implementation within a tour; 

f .6^26 .0030 1 FIG. 5b illustrates various proactive label detection schemes and- 
implicit system driven label detection scheme ; 

toao ^003H FIG. 6 illustrates a process-oriented view of a tour including 
pre -tour 

and post- tour processing ; 

l\6A£Lft .po32 1 FIG. 7 illustrates an exemplary method used for pre- tour 
authoring; 

r^u»^i »0033 l FIG. 8a illustrates an exemplary method used for tour playback; 

f £4 ^0034 1 FIG. Bhg illustrates an exemplary method for tour playback 
specifically 

using a networked remote server site; 

[0031] FIG. 0 illuotratco an embodiment of the preocnt invention in the 
context 

of a guided tour of a comet cry; 

r^^ .0035 1 FIG. 10 illustrates a block diagram of exemplary internal 
components of a 

hand-held mobile device for use within the network illustrated in FIG. 2; 

f^ft ^0036 1 FIG. 11 illustrates an exemplary physical embodiment of a hand- 
held 

mobile device; and 

ro-& ^oQ37 l FIG. 12 illustrates a further exemplary embodiment of a hand-held 

mobile 

device . 



DETAILED DESCRIPTION QF PREFERRED EMBODIMENTS 

roo381 p r gf ftrrflr * ^^imflntfl of the present invention and their advantages 
mav 

[nmc] Tilling Tint,, hn fhn fimirnn.be understood bv referring to FIGS. 1-12. 
wherein like reference numerals refer 

to like elements, *-hmm in innntrnhoriand are described in the context of a 
comprehensive device, 

system^, and method for authoring and providing information to users about 
a : the 

physical wnrl H around the user . In this regard, the oyotcm and mcthodprflaent 
In vention generally provide 

provides information e ythrouah i nfnmnH nainteracfcion with labels, such as^ 
but not limited 

to, machine -readable T~ hlllim i^ nfcifiable labels on physical objects, 
coordinate labels ^e-renresentinq spatial or geographical locations, 

timootamp and time . 

UhPl.g-tom. preferably in the form of timestamps created bv an internal OJL- 
external 

clock-r ni-n whinh source. All labels are treated uniformly as object ^ 
jlQ c a^^pn. or time 

iHpnhif^rft^Phf^. i.e.. each label serves to identify an object, location, 
even t. To simplify the present disclosure, the use of the term 

object 

identifier collectively refers to object, location, or time identifiers. 
These 

object identifiers are more specifically used within the system, in a manner 
to 

be described in greater detail hereinafter, to perform various indexing 
operations such as, for example, content authoring^- and playback, and user 
annotation-r and 

feedback. The flyfl^ffipresent invention is also capable of aggregating object 
identifiers and their associated content into a single addressable 
ttftj^a.taba.se or 

information library referred to hereinafter as a "tour." 

r-ftft^ £0039 1 To provide a comprehensive system and method for providing 
information to 

users about a physical world, and to allow users to record their own 
impressions 

of the physical world, the system preferably f nnnt- i nnnoperates in two modes, 
namely, an 

authoring mode and a playback mode. The authoring mode permits new media 
content, e.g., audio, text, graphics, digital photographs, video, etc .anfl 
various 

other types of data files , to be recorded and bound to an object identifier. 
In 

the authoring mode, the system supports content authoring that can be done 
coincident with object identifier creation^ thereby enabling authored media 
content to be unambiguously bound to an object identifier. Thio oolvoo tho 
pirnhlnm nf mn i nt.n i ninaln other words. 

direct correspondence is maintained b etween physical nhjort-/-, 1 oration-A. or 
timestamp labels and respective media content. The playback mode triggers 
playback of media when an object identifier is arrpfigpri, or detected. In the 



playback mode, the system can also be programmed to accept/_2I-Solicit 
annotations and/or feedback from a user which oan£ a be recorded and further 
unambiguously 

bound to an object identifier. Annotation and feedback arc both may frft 4n fchQ 
jfiprm of 

user responses to objects ^encountered. The difference between flMlPtflfc l pn 

_ ^ _ . . . . 

feedback is fairly small in that the user generally owns £feeor yepaing T^qflts 

to 

annotations while feedback is typically owned by the person who solicited the 
feedback. Also, feedback coulda iy. be interactive.,, such as^ a user responding 
to a 

sequence of questions. 

[ 003^] Tu r n— g — ^ i T?ir . n nnrt thn00401 The following description 

arniB intended to provide a brief, general dcocriptionoyerYiew of a 
suitable computing environment in which the invention may be implemented. 
Although not ™ t ,it^h ^ l^ted as such, the invention will boi a described 
in the general 

context of computer- executable instructions being executed by ope qr ffl o re 
fl fr gt^ributed computing devices. The computer- executable instructions may 
include 

routines, programs, objects, components, data structures, e^aad the like that 
perform particular tasks or implement data types. The portable 
nnmpnt-innMnrftover, the present 

^n^nnn o n t j n y ^n , t , fr n fl W\Y hft operated by mobile users may include through the 
implementation of portable _ 
nnmnutina d evices «ugh ^. but not limited to, hand-held devices, voice or 
voice/data enabled cellular phones, smart -phones, notobooko , nptefrOPK 

t :ablet s^ "we ar^le^ computers , personal digital assistants (^PDAs^.) f or gpeffifll 
put-nose built device - These devicea mav he configured with or without a 
wireless network interfac e, purpooc built devices, etc . The 
-j nvnnt- 1 rm inventive concept may 1 aloo be practiced in 

distributed computing environments where tasks are performed by computing 
devices that are 1 inVp d r p referably through a wj.rele aa communications network 
and 

where computer -executable instructions may be located in both local and 
remote 

memory storage devices. The remote computer system may include servers, 
minicomputcro , 

mainframe computers, — storage ocrvcro, databaoo ocrvcro, etc. 

[ nmo] Mnrn npn n i f i nn 1 1 y O 04 11 A ccording to a preferred embodiment of the 
invention , FIG. 3-1 illustrates 

portable computing device 1Q5 in a network architecture ^-in which a tour 
server 

side is coupled to a client side via a^-wireless distribution network 209. 
mvi-iin hV.n m-TT-nl nnn us. wireless 

distribution network is preferably a voice/data cellular telephone 

network , 

however, it will be apparent to those of ordinary skill in the art that other 
forms of networking may also be used. For example, the network can use other 
forms of wireless 

transmission nunh nn RF networka based o n r but not limited to, radio frequency 
( "RF" > . 0 02 .11, 



802.11 standard, and B luetooth, oto . in a Wirolcoo Local Area Network (WLAN) 
— 1 T.n.i 7Mrn^f ftf ejramole . a wireless local area network 

Network (WPAN) , oto. 

( "WLAN" ) or npr fiQ nai local area network CWPftN"). 

fQQ30 0042 1 Connected to the wireless distribution network 2£9H§, on the 
client side of 

the network 200 are one or more mobile users an a which nanwho mav roam indoor 
and/or outdoor 

locations t o thereby move among pinml -i r.y nfoi^e or more objects 2G±2JU. in 
the physical world. As will 

be described in greater detail below, tfee— locations 108 and/or objects 3£il&2 
in the 

physical world can be represented by one or more machine readable px~ 
^ofon ftif labia o bject identifiers, such as, barcode labels, RFID tags, IR tags, 
Dluc tago 

•{•Bluetooth readable ^asi. analog to digital convertible tags: and/or further 
a ««o e iated with human identifiable text, location coordinates ("labclo in 
t-Hn .-M-riM nr . and timestamos. In thio regard, timootampo 

Timeatamna generated bv internal clock 109 Oil mobile device 1Q5 can serve as 
labels efiia their own right or can be considered to be qualifiers to the 
media 

content bound to an object or a place. By way of example only* media content 
qualified by a timestamp we»ld£guj^ be information pertaining to a mountain 
resort 

location where wi ntnr w inter information could be different from Gummorpf^mmfir 
information. 

f QQ4Q 0043 1 Location coordinates 4-108 representing, for example, latitude, 
longitude, 

and optionally aUimri M may be . are determined by a location determination 
unit coupled 

with the mobile device using signals transmitted by GPS satellites or other 
sources. M t "r ni f -- 1 y + ? fr rt tl fr° T ^H iffi6nta - location of the mobile 
device is determined bv 

other conventional location determination schemes, m vet anpttar alternative 
ffnh ^niflnt. the location coordinates can be provided afehy. a remote server , 
and any 

mobile device requiring such data can addrcoor eceive the location data 
request tn n from the 

networked remote location server. This is especially useful when the mobile 
device does 

not have location identification capability, or in indoor facilities where 
GPS 

satellite signals are obscured. The location of a mobile device conncetcd to 
an indoor 

WLAN acccoa point can be approximated by the location ocrvcr connected to the 
WLAN, by conoidcring known location(o) of wirolcoo acccoo point(o), the 
signal 

strength detected between mobile device and acccoa point (o), and poooiblo 
uoing 

additional opatial information about the geometry of the cnclooing building 
space . 

ro041Q0441 To read information from the object identifiers, each mobile user 
2 0 8 io equipped with a personal mobile device H-nr-i nrj ins com prises 



capture circuitry that is adapted to respond to ^-location 

i^lfc "**** n!phnrf. ;] 05 attached to physical ohHect 107, Capture circuitry 

barcode reader, RFID reader, IR port, Bluetooth receiver, GPS receiver, audio 
receiver/ 

touch- tone keypad, *^?PY ^alog to digital transducer than can transform 
label 

information to digital data, or anv combination thereof. In the networked 
environment, the p ersonal mobile device nnn mn^QS runs a thin qr applSfc 
client system 26^104 

with input and output capabilities while storage and computational processing 
takes place on the server side of the network. The client system may include 
a 

wireless browser software application such as a ff^eles^ application protocol 
("WAPM browser, Microsoft Mobile Rvplnrpr^^.RTM. . and the like, and 
support 

communication protocols with fhn implemented on anv type of server well known 
in the artaart. 

such T , T?m uttd nt- n Tn . but not limited to, a WAP or hypertext transfer 
protocol ( "HTTP") based 

non networked applicationo, — the poroonal mobile device 207 can contain 
additional local indexed otoragc 205 in addition to the client oyotom 204 
whereby all proceeding can take place within the pergonal mobile device 207. 

server i 

mna2 0045 1 In a networked environment, a— tour mnv be!03 is transported via 
Path 113 between a- 

remote server **^ti4 and mobile device 105 by a wired connection or a 
wireless nrmnonM nn network 115. In the wiredgpecj fig 

^-^ wWe tour application 104 is implemented on a phone, the application 



may. 



the context ofi a yoi 



CVoiceXM L'M browsftr or locally on the device. Index table repository 116. to 



re^otely d ac^ the^multi^ 1 ^ ggg *^ CTt CT 

1-hn f.niir nn ri collection associated *U4^wifch an obiect i< 
locally resident 

on the device or downlo aded or streamed via path 113 with the aid of content 
proxy 117. 

rQ0461 Tn an alternativ e AinhnHiment, a wired network may be substituted for 
all 

or part of the wireless network. For example, transfer of tour 103 m ay be 

done directly 

implemented by a modem connection (not shown) between fe^emobile device 105 
and a— 

remote serve r 114 or indirectly using a hoot computer ao an intermediary— 
Bvnmplnn system 100 using data 

paths 102 and 101. Moreover, a tour mav be authored on a hOE 



client aufchorincr system 100 and 



>rred to the device using data 

101 or uploaded to the se rver using data path 102 for subsequent download 
later 



are described in 



n ^^ f . Fur ther examples of transferring a tour from a 

mobile 

device to a host computer via wired oonnootior 
greater riot-nil ndetail 
below. In tho 

wirclcoo caoG, opooifioally in the caoc of the tour application being uocd on 



phone, the application may run both remotely in the content of a voiooXML 
browocr or locally on the device. 



In the remote server playback case, the connection between fehe- 
server.jyjt and 

the phonc ^hile device 105 need not be held for the duration of the entire 
tour. gk^For 

exa m ple, the server nnnl d can maintain the state of the of tho last rendered 
position in the 

tour across multiple intermittent connections permitting the connection to be 
re-established on a need basis. The state maintenance not only avoids the 

user . 

having to log back in with a username /password, but puts the user right back 

to where he wao 

the last location in the tour, much like a compact disc ("CD ■) plaYSr 
remembering the last played — s** 3 - ftp gp - If mobile device 105 it 
^^i^^a^^y^ 

^pted cel^yi^r phone, the server can use the caller's phone number to 
identify 

the last tour the user was in. In certain scenarios where the caller's phone 
number cannot be identified, a user would be prompted for a uocmamcusername 

and . 
password and would be immediately taken to the last tour context. This 
functionality not only saves on the connection time costs, but also is 

effective . , 

for certain applications such as a tour implemented for providing driving 

directions using VoiceXML. 



[4e44£Qi&] For tour authoring and publishing purposes— fefee^ mobile device 207 
«*U»to.ms comprises a 

have a universal serial bus ("U SB") connector so that the mobile device and 
can be directly 

connected ^* P ath 101 to a— host ^^r n1 ~ &r — 1Q1 - Tn an alternative 
embodime nt where 

the personal mobile ^^"^ rlnrievice does not have a communication 

link, ouch ao an USB connector, a ocheme for tour retrieval (i.e., 
nplnnriina u pload of the tour to 

a host computer^- can be implemented using a conventional data output , such 
as r an output connected to the microphone input of a PC, Although 

such 

output. Though thioa, scheme mnnUnp a y result in some audio quality 
degradation in the re -recording 

process, it would serve as a safe-backup of valuable content on a PC. When 
sequential playback is initiated in a particular device mode, callcdref erred 
to as an 

nTTpinnd m iyh n ni mpload playback mode , " the index values of a tour are sent as 
specialized tones 



whose frequencies are chosen so to not collide with human speech. The output 
of the hcadphonco io connected to the microphone input of a PC. Special 
software running on the PC recognizes the alphanumeric index delimiters 
between 

content and regenerates a tour. The alphanumeric indices values could 
represent 

normalized label values^, such as^, timestamps, barcode values, or coordinates. 

I\ft645 -0049 1 To provide for the authoring and/or playback of media content 
related to 

»n« object identifier or a plurality of object idenUfifirff agffQC i fltfifl with a 

ar-tour, a-personal mobile device a-ft3^105. examples of which are illustrated 
in FIGS. 

10-12, preferably includes object label decode circuitry 1002 that is adapted 
to 

read/respond to barcode information, RFID information, IR information, 

4^^direct or x ^ ^ 

^ r ,f npnnnh fnin^rfigh fobfained frqift an analog to digital transducer) text 

input, geographic 

coordinate information, and/or timestamp information. The object label decode 
circuitry 1002 provides input to a^tour application 1004 resident on the 
personal 

mobile device 0^ ^105. The tour application, which will be described in 
greater 

detail below, generally responds to the input to initiate the authoring or 
rendering of media content as a function of the object label read. For 
playing 

the media content, the personal mobile device 2 07 may include one or more of 
^jOR romnriges video decoder 1006 

associated with a-display ^-AAa-1008. and an audio decoder 1010 associated with 
a 

speaker 1012. Display 1008 may be a visual display such as liquid crystal 
display screen. *^In m f*l*- ftraativ » fnflff^*P Anfc " the device mayfiaa function 
without a 
Y ftsual display . 

r^uw ;0050 1 For inputting information which may be bound to an object 
identifier, the 

personal mobile device nnn ™-»y n1 nn -inninHn ms com prises a means for 
inputting textual information 4 

via, e .g., -a keyboard im<D . 1014, a pointing device ouch aoin the form of 3 
p^n fnpfc shown) . ^ 

touch sensitive screen whic h that, is part of fetee-displayr 3-008? means for 

information -( -via, e .g., a^-video encoder 1016 and video input 1018-hrj. and/or 
pneang for 

inputting a udio information 4yj^ t== e.g., an-audio encoder 1020 and microphone 
10331 . 1022, 

touch- tone Hin-t-ons . such as, dual tone Tmilfci frequency CLDTMF") buttons 
(not 

l) for phones. Variouo 



r00511 Referring fco FIG. 11, personal mob ile device 1100 comprises media 
content 

control keys such as, mmmpi n piay ^ /ato p 1101. record- 1103, reverse-^ 
1105, fast forward- 



1104. volume ™nMrn1 . nt- n . controls 1110. and various other operations can be 
provided for use 

in interacting with media content. In this manner, the various control keys 
can 

be used to selectively disable device functionality in certain device modes, 
particularly playback mode, using hardware button shields, device mode 
selectors, or embedded software logic. Personal mobile device mav 1100 
further comprise one or more of the following: an audio input, e.g.. 

microphone 

1102: audio output, e.g.. speaker 1106 or headphone output 1109: barcode 

and/or 

RFID scanner 1108: display 1107: power switch 1111: battery slots 1112; and 
device mode selector 1113 for alternating between authoring and playback 
modes . 

f00521 Referring to the alternate ve embodiment depicted in FIG. 12. mobile 
device 1200 comprises media content control kevs such as. plav/stop 1211. 

1208. reverse 1201. fast forward 1209. volume controls 1216. and various 
other 

operations that can be provided for use in interacting with media content. In 
addition, the device 1200 comprises audio prompt response buttons 1203 and 
1212 

for responding to audio Questions posed bv the device. Also the device mav 
have 

tour based operations, such as. new tour creation button 1204. tour 
navigation 

1205. tour/slide deletion 1213. Personal mobile device 1200 mav further 
comprise 

one or more of the fol lowing: an audio input, e.g.. microphone 1202? audio 
output, e.g.. speaker 1206 or headphone output 1215: barcode and/or RFID 
scanner 

1207: power switch 1219: battery slots 1220: removable storage 1214: USB 
connector 1217: power for battery recharging 1218: LED 1210 for visual cues. 

r0047 r00531 The mnhiln nnrnonal dnvinn 207 inventive concept can be 
implemented on any type of c omputing device, 

ranging from n pornnnnl nnmpnt-.nr. nntnhnnk . tnhlnt. PDA, nhonaoxistina 

purpose-built . i.e.. cuf**™ ^Hft f device. SincoBecause the^a, tour application 
does not 

mandate the implementation of all object identification schemes, a-mobile 
personal 

device ^&^105 may implement the label identification schemes most suited for 
the 

particular device capabilities and usage context. Also, a— mobile personal 
device W 

105 may only support the authoring and/or rendering of particular media. For 
ovampi cx f for those mobile devices 207 that do not have the resources^ 4e.g., 
a 

resource -constrained phoned to support the full capabilities of the tour 
application, a tour application proxy could be built for the device, and the 
resource intensive processing can take takes place on the server side. 
Further , the 

im pleme ntation of tour applic ation pr oxies 116 and 117 is done based on the 
storage and computing resources of th e device. For example, in one 
embo dim ent r 



index tab l e me i» ^mn ? «ed of object identifiers that are 
but 

multimedi a content -nil action 117 is remotely remount. In another 
indav table 116 i» also remotely resident, i.ft., thfl proxy directs al l 

normalized innut o bta in ed from a label detection scheme to remote flerver Hi. 
The latte r emhndi menr mav ha preferred on resource conntrained dftYiCOB BUCfr 



cellular nhnnae. For » device that has enough computing and Storage 

reso^rc^a . 

Hnfch comp o nAnfcfl of the tour, index table repository 116 and multimedia 

content 

collectio n 117 can be locally resident on the device, 

r-ftA44 ^Q054 1 Turning to the tour application, -^he tour application 1004 
preferably 

includes executable instructions that can create and modify a tour tree 
e^niri-iirp U. which is d iscussed in greater detail below^. for performing 
various tree 

tour o perations such a a, but nofc ^mifced to- tree traversal, tree node 
creation, 

tree node deletions, and tree node modifications. The tourInd ex . tflfrJifl 
3,3. King 

either loc ally resident or remote on 



Ann 

flfti-Yftr. Tour application 1004 aloo supports fcke-authoring, the playback, 
annotation, and/or 

feedback of a tour. The tourl flux, application 1004 may also support thft 
transformation of 

a tour from one particular format tranof ormationo of a tourfro anther , it 
will be understood that tho tour 

application 1004 can work in connection with a proxy to perform these 
functions . 

Still further, the tour application 1004 can be a stand alone module or 
integrated 

with other modules such as, by way of example only, a navigation system or a 
remote databaoo . Tn i-h-in ^n this 

latter instance, while the navigation system would provide the details of how 
to 

get from point A to point B, tho tour application 1004 could provide 
information 

pertaining to locations and objects found along the path from point A to 

point 

B. 



[0040] At the ocrvcr side of the network 200, the ocrvcr aide io preferably 
implemented ao a computer oyotcm which io connected to the wirclcoo network 
209 

by one or more acccoo acrvcro 21G. The acccoo ocrvcro 21C may be a WAP 
gateway, — 

voice portal, HTTP acrvcr, CMSC (Short Mcooagc Service Center) or the like. 
Additionally found on the ocrvcr aide io an object information ocrvcr 210, an 
optional object naming ocrvcr 200, and an optional location ocrvcr 211. Tho 
object information ocrvcro 210 contain an indexed collection of multimedia 
content, which may rcoidc on one or more external databaoco (not 
illuotratcd) . 



Tho object naming oorvcr 200 acta ao a maotcr indcxcr for the object 
information 

ocrvoro 210 and can be uocd to opced up acccoo to data. The location ocrvor 
211 

can bo uocd to compute the location of a mobile pergonal device 207 baocd on 
data received from the wirclcoo network 200 or from outoide oourceo. Tho 
location oorvcr 211 can further work in connection with a map oorvcr 212 and 
with a floor plan aorvcr 213 wherein tho floor plan ocrvor 213 can bo a 
digital 

ropooitory of building layout data. Tho oorvcr oidc may aloo include an 
authoring oyotem which can bo uocd to add, delete, and/or modify media 
content 

otorcd in the information oorvcro. It will be appreciated that the variouo 
computoro that can bo uood within the ocrvor oide of the network may 
thcmoclvoo 

be connected to one another via a local area network. 

r<&gjie -0055 1 To provide information to a user via a mobile personal device, and 
as 

noted previously, the system may use the concept of a "tourj 1 which can be 
considered to K n nr . nr^nrn^ 1 i nh nf nl i Hon an ordered list of media content 
that are indexed by object 

identifiers created from text otringo, phyoical object labclo, eoordinatco of 
gomrjTrnph-innl InnnMnnnfor example, text strings, physical object labels , aftd- 
timootampo roprcocnting temporal evento. In thio regard, — a olidc io an 
ordered liot of media content which can 

coordinates of geographical locations, and timftBtamps representing temporal 
ftvftntg. In this regard, the media content mav optionall y further contain 
annotations and feedback. Annotations and feedback are also lists of media 
content. Media content can further be considered to be an ordered list of 
digital content in text, audio, graphics, and/or video stored in various 
persistent formats 311 such as, by way of example only, XML, PowerPoint, 
GMIL, etc. 

synchronized multi media integration language ("SMIL'M . and the like, as 

illustrated in FIG. 3b. The olidco in a tour may be optionally 
aggregated into nodoo called channolo. — 

r&e-£3 r0056 1 In ft ^a particular embodiment^ fehea, tour is implemented as a 
collection of 

multimedia digital informatio n library , where the multimedia content is 
indexed by 

normalized labels^ -fi . e . , object ir^rn- i f i^T-.cO-^ generic to two or more 
interpretation schemes . stored in index table repository 116. The digital 
information includes audio files, visual 4ma^egraphics files, text files, 
video 

files, multimedia files, XML files, SMIL files, hyperlink references, live 
agent 

connection links, programming code files, configuration information files, 
other 

data files, o r a combination thereof. Various transformations can be 
performed 

on the multi-media content. ^ nm r^ nfr -> ^nnnf nr-m,->M on in whonFor example. 
recorded audio is transcribed into a 

text file. The advantage of content format transformations is to allow 
accessing 



the same tour with mobile devices of different capabilities and^ox according 
to 

user preference. An example of this is accessing a tour using a voice only 
cellular phone or accessing the same tour with a PDA with display 
capabilities. 

Uwi a0057 1 The aggregation of media content can be done to any depth as 
deemed 

appropriate to the application context. This is particularly illustrated in 
FIG. 

3a.,. which depicts an exemplary instance of a tour in the form of a tree flflftfl 
structure. The nodes of the tree are the tour node 301, the channel node 302 # 
the slide node 

303, 4=4ieand media node- 304. Tth hhn mnmpl Particularly, media node 304 

flfflroriaftg <?r Units t° 

fcAxfc. aud in. video, graphics, and other data. Slide node 3Q3 POintS tQ QW Qr 
media n Q de fl 304. channel node 302 aggregates one or mora slide nofleg 

303. 

Thifl aflar p aation is to facilitate logical grouping of content within a tQUr. 
For 

Avamnle. in a museum- specif ic tour, all exhibitfl within the Science Section 
mav 

he grouped into a channel 302. Tour node 301 aggregates all Channel ny^es 3Q2 
into the complete structure that constitutes a tour. In the exemplary 
instance 

of a tour fibnw n in FIG. 3a . index table 305 is associated with the tour 
tree . 

The flexibility an d richness of the tour data structure enables various 

iur 310 between different file formats 311 as illustrated 




raaj ^OOSS I Index tables 305 are particularly used to gain access to the media 
content associated with a tour. In this regard, an indexing operation, 
performed 

in response to the reading of an object identifier, can result in a tour, 
slide, 

or channel being rendered on a— mobile personal device 207 .105. As noted 
previously, 

the tour, slide, or channel can be provided to the m obile personal device 
3^105 from 

the server side of the network and/or from local memory, including local 
memory 

expansion slots^ 

r&&54 -0059 1 The nodes of the tour hierarchy can contain information 
appropriate to a 

given application which can use a logical structuring of information without 
regard to file format specifications or physical locations of the files. 
Accordingly, there may be several physical file implementations of a tour 
and, 

so long as the structural integrity of the tour is preserved in a particular 
implementation, transformations can be done between different file formats. 
However, it is cautioned that, during a transformation, some media content 
types 

may be inapprnpriat.fr/ . or " lost" since the destination mobile personal device- 
W may 



not support some or all of the media content in a tour. For example, a mobile 
personal device ™i TT ^ h nnwithout a display apd only flwUft gapab;Uit;;i l eg 
would be limited 

to presenting tour media content that is only in an audio format. 

f4W ^QQ60 1 To author a tour containing information about physical objects, 
locations, and/or temporal events H ^ , ^^ff^v filv referred to as 
^entitiesJL) in 

the physical worlds the entities are labeled whiefe&ifck labels fchafc are 
treated 

uniformly as object identifiers. The object identifiers are stored within the 
system and media content for an entity is bound to its corresponding object 
identifier. When assigning labels to objects, generally illustrated at stage 
401 

in FIG. 4, objects that do not have a preexisting label are provided with a 
customized label. Objects with preexisting labels can include items that have 
UPC coded tags. Example of custom labeling would be the labeling of a picture 
in 

a photo album or a paragraph in a book. It will be appreciated that, even for 
objects that have preexisting labels, custom labeling maysaa be done in- 

n»Trt-- l -in nirmimnfnnpnnif desired . 

The remaining stages illustrated in FIG. 4 include stage 402 where 
objects/object identifiers are bound to media content and stage 403 where 
optional feedback and annotations can be bound to objects/object identifiers. 

r ^^ -0061 1 To label geographical location, the concept of a "label in the 
nivn in location coordinates are introduced. In as- 
authoring mode, an authoring device, such as a personal mobile device 207, j_ 
determines its current location coordinates using a— GPS or similar 
technology, or 

using information available from the wireless network. The computer 
coordinates 

may then be used as the object identifier for the geographic location. The 
author may bind media content to a "label in the air" coordinatea t he same 
way as any other label . 

Furthermore, the usage of coordinate data does not require the exact 
coordinate 

to be available to initiate playback of the media content bound to the 
"label in the air 

inordinate . ^ Rather, a circular shell of influence may be defined around the 
coordinate that can trigger playback of the media content, jmedia content. For 
simplicity of 

authoring, it is preferred that the shell of influence be a planar projection 
of 

the coordinate thereby eliminating the need to consider altitude variations. 

r.&a5 ^oo621 It will be further appreciated that various concentric circular 
shells of 

influence may be defined around a coordinate label which ohcllo of 
inflimnn o and can be bound to unique media content. 

ptedia content. In this manner, entry into these various shells can trigger 
audio 

and/or visual content authored explicitly for that shell. This can be 

particularly useful in gaming applications such as, for example, a treasure 

hunt . An example io uoing color ao an indicator of diotancc 

from the labeled object io to dioplay "cold" blue on the mobile device when 

^he- 



troaouro hunter io far away from tho object and gradually turn tho dioplay 
"warm" rod (ao getting oloocr) to "red hot" when the treaouro hunter roaoheo 
the 

object . 

r Temporal events require no further labeling, i.e., the timestamp 
can 

serve as the label itself . In this regard, timestamps can be used to label 
both 

periodic and aperiodic temporal events. Furthermore, even when labeling 
aperiodic events, timestamp labels can have an artificial periodicity 
associated 

with them to serve as a reminder of past events. a**m an o^^-i^anh of t-ha 
invention, an internal clock within a-personal mobile device 207 nan bolQ5 is 
used to check 

the validity of timestamp labels which, when read and if valid, can initiate 
content rendering in playback mode. When using timestamps to label aperiodic 
events, the timestamps are used as secondary labels to a primary label such 
as a 

physical object label or location coordinate. Such labels are thus identified 
as 

a consequence of identifying the primary label. 

f.frft5J »0064 1 Text strings can directly serve as labels for indexing media 
content. Mor 

in pnnnihln that thnoxample, text nt-.rino wnnstrings mav be the Output of a 
npnnnh rnnnrrni nnr By wnvtranaducer that can transform anv 

non-digital data into digital data, for example, a text string or anv other 
computer specific data type that can represent the digital data, Bv wav of 

e€— further example, an instance of a tour can be a hierarchical set of markup 
language, e.g., XML or hvner-text markup language ("HTML"), pages combined 
with 

one or more index tables. With the addition of index tables and ordering of 
the 

pages, an existing web site could be implemented as a tour where all indexing 
is 

done using text strings. 

rfr&6 ^0065 1 ^Pfee4 labeling scheme for physical objects coul dg an range from 
manually writing 

down a code on an object to tagging the object with a barcode, RFID tag es^ 
IR tag^ 

or anv conventional type of identification means. For scenarios that need 
custom 

labeling, the labeling can be done in any order regardless of the labeling 
scheme being used. This eliminates the need to maintain an extraneous order 
between labels and objects which, in turn, eliminates errors in the labeling 
process . 

f nnsil Th n OQ661 I n embodiment of the invention, data structure 
representation for a 

normalized label could boj La a variable length null -terminated string. 
Wb^Alternativelv it 

could be any data type that can represent the digital data that was retrieved 
from the label, the retrieval being followed bv an optional transformation of 
non-digital data into digital form. For example, when a barcode label is 
scanned, the scanning device returns the label in a device specific manner, 



which is then transformed by the normalization process into a null terminated 
string. For example^ if the value encoded on the barcode label was the UPC 
code 

of a particular product "Altoido" brand peppermint candico , after the 
normalization^ it would become a PiVTtflfrifi string, 

otring of the form "0502 8 000200." Note that the normalised otring valuo doco 
not 

such as, "05928000200." which does not reveal any information about how the 
value was retrieved — because normalization strips out all information 
about the 

particular label retrieving process. These normalized or generic strings, 
also 

referred to as object identifiers, are then used as indices for organizing 
authored content . 

f-&Q63 -0067 l During content authoring, since labels are normalized into object 
identifiers, multiple labeling schemes may be used to access the same piece 
of 

media content, provided the data encoded by these labeling schemes 
yi nl dv ields the 

same value after normalization. For example, an object can be labeled by 
associating a UPC text stream therewith and media content bound to the object 
can be retrieved by entering the same UPC text stream or by scanning a UPC 
bar 

code corresponding to the UPC text stream. In a further example, a coordinate 
obtained from a GPS type device may be embedded into a barcode label, an RFID 
tag, or even etched into an object. Thus, in playback mode, deocribed below, — 
a personal mobile 

device ^105 with any one of the label detection capabilities, e.g., barcode 
reader, RFID tag reader, IR port, digital text or opooo h analoa to digital 
text 

transformation capabilities, can be used to retrieve media content bound to 
the 

object identifier corresponding to the object since, in this case, the 
information that is embedded into the different labels is a normalized form 
of 

label data, namely, the coordinate. For multiple labeling schemes to index 
the 

same object the data in multiple labels , the scheme should be such that they 
all 

result in the same normalized value. In the above example, the barcode label, 
and the RIFD tag, embed the same value — . e.g.. location coordinates. 

f-&&& ^0068 1 Just as multiple labeling schemes result in the same normalized 
index 

value (referred to as the object identifier) , multiple distinct object 
identifiers can refer to the same object. An example can 
illuntrato illustrates the difference 

between multiple labeling schemes used to yield the same object identifier, 
and 

multiple distinct object identifiers indexing the same object. Consider a 
street 

with andan embedded RFID tag. The coordinate values returned by a GPS device 
could be are 

embedded into the RFID tag. Content could bej ^g, authored for the normalized 
value- -the 



a lso create a text-string label for that street name 
coordinate. A user may also create a 

and . - that labe l to the same content. When a user of 

S 1M or . RFID reader. Alternatively he may read f^^^^T' 

result^ a different labeling value that indexes the same content. 
W^BU, Further, if the device only has location determination capacity 
texfinput mechanism, the location of the user could be used to narrow down 

Sect identifier search space. ^^^^^-^^ 
^^^^^ it can he used for automatically listing 

all objects in the proximity of number of obje cts, the 

the user. In those scenarios where there are Jl ^ Qf ^ gtreet name 
culled search space could help the user oy 

as _ „ f . he device with keyboard input scheme) , or 

speech v . . . hv the user . In this scenario, two object 

ssssis rruseHn ^ssxr-s » - — ■ 

oTthe o bj ect identifiers (location coordinates, is used to aid the detection 
IL other (the street name text string) . 

[MWlt2SJ > special case of multiple labeling methods being used to refer to 
fame media content is the functionality to index any tour with an ordinal 
Sue of the content, the implicit ordering of content present in a tour. 
Bering provides an alternate way to get to authored content regardless of 
noised labeling method. This is a special case because the normalised 
^digital text string representing the ordinal index of the content whrch 
5 be the same as the normalised index type explicitly used during 
authoring. coordinates being used as the normalised 

vaLfcafbe index value for that content. 

TO access and/or author media content, a label identification 
pTrformed'as illustrated in no. The outcome of the label 

ProcesfuanWct identifier that can be used for indexing. As 
^"'identifier is independent of the label type. Furthermore, as noted 



above, different kinds of + - cn V,^l * nnut «ch— ■ 501 can be embedded 

™fcrieve different types of labels 5**5fl2. and the normalization process 503 
yields 

a normalized index value. riafca r»mrin>d from the lahftl normalization 

RfH mav b e ^rA^ntqfl fry computer support data tYPft and flgf, liffiltgfl fc Q fl 

alphanumeric string. 

fAA ^p072 1 In the authoring mode, *4^1abel identification of the labolo is 
done proactively by the 

user either manually or with the aide of an apparatus, such as a bar code 
scanner, optical scanner, location coordinate detector, and/or a clock. An 
object identifier can be used to generically represent one or more of these 
identified labels. Specifically, an object identifier can be used as a 
normalized representation of different labels and, thereby, can serve the key 
purpose of allowing different labels to uniformly index media content in a 
manner that is transparent to their underlying differences. Furthermore, as 
noted previously, since labels are treated in a normalized manner, it is 
possible for label detection to be performed differently during the authoring 
and playback operations. 

teA6 ft0073 1 To maintain the association between an object identifier and media 
content for an object, an ^drvrnd Hnhihnnn index table is created during the 
authoring mode of 

operation. When a label is identified and an object identifier created, a- 
search 

HI i s done for the object identifier in the databaoc . j nd 6 X fcafrlfl y e P Q g l t Qr V 
116. If the 

object identifier is not already in the databaociflflex table repository 116 
the object 

identifier is added to the ^^^nn index table repository 116, As an example 
only, the databaoo 

index table repository 116 can be implemented using index tables and flat 
files, 

relational or object based database systems, naming and tfta liK Q * 
directory ocrvicco, — cto . 

• f -6* ^0074 1 Once an object identifier is identified within a databaoc, inflex, 
table repository 

116. media content can be mapped to the object identifier. As noted 
previously, 

the media content can be in one or more formats including text, audio, 

graphics, t 

digital image, and video. Multiple media content can be associated with the 

same 

object identifier within a d . -n-nhnnn index table repository 116 and can be 
stored in one 

or more locations. To remove errors in the indexing process, such as 
associating 

media content with the wrong object identifier and, accordingly, the wrong 
object, when a new object is identified in the authoring mode, the system can 
create a new entry in the ^M^nninto table -renoBitorv 116 and immediately 
prompt the 

user to author/identify media content that is to be associated with the 
object 



identifier. This coincident object identifier creation and 
author ing/ ident i f y ing 

allows media content and object identifier binding to occur nearly 
instantaneously . 

r^A? ^0Q75 1 The advantage of the labeling and media content scheme described 
above is 

particularly seen in practical applications such as, for example, home 
cataloging situations where picture albums, CD collections, book collections, 
articles, boxes, ***^-*nH other articles are organized. also finds use in 

commercial contexts, both small and large, where a vendor might wish to 
provide 

information on objects being sold. An example of a small commercial context 
usage is an antiques vendor labeling his articles and/or parts of articles 
and 

associating media content therewith that might explain historical 
significance . 

In this regard, the objects can be quickly labeled in any order and have 
content 

quickly and easily associated therewith. In a larger commercial context, a 
vendor can author daily promotions and sales information by scanning a label 
associated with an object and associating media content describing the 
promotion 

and sales information with the object. 

(•Q071 Q076 1 While *-v . n H^nhmn ^ ^ table repository 116 can be created using 
a host computer, it 

is preferred that Mm Hnhnhmn^to fable repoaitorv 116 be created using the 
mobile 

personal device 4W7 ^105. To this end, the mobile personal device allows the 
user to 

read the label and author the content that is to be associated with the read 
label. The mobile personal device ^&^105. or the server side components, 
will then 

automatically map the content and the created object identifier to each other 
within ^t-nb-mn table rapo^torv 116. It will be appreciated that 

this makes the 

binding of coordinates particularly easy since the content author can 
directly 

create content to be mapped to the coordinate at that very location. A 
particular example of this would be a real estate agent creating a tour of a 
home while touring the home. It would also be possible for a potential 
homebuyer 

to author feedback which can also be mapped to the coordinates as the 
potential 

homebuyer tours the home . 

rQ0771 T he process for authoring a tour is generally illustrated as steps 
612-614 in FIG. 6 (pre-tour 611 being performed with the assistance of aR- 
authoring 

tool 615) and steps 701-709 in PIG, 7. Authoring process 611 begins bv 
labeling 

(step 612) objects if thev do not alread y have a label or require application 
spe gific labeling. Steps 701 and 702 correspond to these steps for an object 
that doe * f not have a label. The labe ling of ob-iects (step 703) can be done in 
anv order. Subs e quent to the labeling, in the obnect cataloging (step 613), 



index tabl e <. n-r fta h e d ii B ina the lsbfll indices stained bv fann i ng the 

qb~j Qct 

labels and normalizin g the rehriftved labels (step 704) ■ flimultanQQUfi tO thS 

^ f ^^nn- content , js authored and bound to thes« indices (fffrftP 7Q51 * 



The 

authoring nr Qeeafi * 0" , lfl "Q^e bv authoring tool 706 that is reside 
mobile dev ^A. The fjn*1 step in the tour authoring prPCftffft involves 

publishing , _ ^ 

it. which could ran ge from saving it in local storage or downloading tQ a 
mobile 

device or u nloading it to a server. The storage Choice WOUld be determined bY 

FIG. 7. Furthermore, an£jaa author nnn nbnnnoof the tour. An author chooses to 
make some or all of his tours 

priv a i- P or public (step 707) . A private tour does not mean that it cannot be 
stored on a or-v^r— hut rather refers to generally that only Particular 

Public touro aro open to public, poooibly at a price. It io loft to 

fuirrhoffisftd wmm my view the 

riinnmM'nn nf th nmedia content nmnfnr frypigallv stored in a private secure 
storage ( fl ^n 708). User authorization and data verification can be Perfora t ed 
using conventiona l techniques- Moreo^r a ecuritv of the media Content Can be 
enhanced fr y implementing one or more cryptographic tftChniOUeg , Wish ftB i fr i t 

■fal or asymmetrical encryption, digital signatures . 

lere security is not of concern. Public tOUEB Can b3 

y , c fste P 70<n. m an embodiment of the invention, access 



£o_ 

the tour is granted upon a user's Pa^ 

Uvo^a -0078 1 still further, browsed web pages can be aggregated into a tour 
since the 

browsing process creates an ordering of content and an index table with the 
links that were traversed during the browsing— f» Moreover, it is also 

nnnnm'vnhl OPOSSible 

that all hyperlinks in the pages visited could be automatically added into 

the . 

index table-)-. The browsed content can then be augmented with annotations and 

feedback which are bound to indices accessed in this browsing sequence. Thus, 

playback of one or more tours or conventional web browsing can be treated as 

an 

authoring of a new tour that is a subset of the tours and web pages navigated 
in 

playback mode. This functionality is very useful to create a custom tour 
containing information extracted from multiple tours and conventional web 
pages . 



To playback media content that has been mapped to an object 

identifier 

within -> rqnf-.hnnn a n index table repository , the system determines the object 
identifier 

for a read label, searches for the object identifier in a databaocindex table 
repository , retrieves the media content associated with the object 
identifier, 

and sequentially renders the media content on the personal mobile device- 
207 . A This 



is generally illustrated in FIG. 6 as steps 622-624 related to the tour 
process 621 

and as steps 801-804 illustrated in FIG. 8. The first step in tour playback 
is 

the label detectio n, ff ^ epa 622 and 8Q1) • The normalized label is then used to 
index an index tab] ft rpnoaitorv. If the index is found (step 802) if. rftSUlta 



retrieval faten 623 1 of media bound to that index during authoring Stage and 
renditio n of the relieved media (aten 804) . If the index is not found, a 
tvnical action would he to report an error to the user (step 803). The tPUf 
mav 

he also a uthored to provide alternate index lookup schemes to find an 

index auc h *a. for example, an index search in select URLa. Tf f.hfl index Ifi 
found, then that in d ex can be added to the tour's index table repository and 
the 

content c a n then become nart of the ordered elements Of the tOUr. Subsequent 

the rendi t ion of the retrieved media, the tour mav have been authored tQ 

Bolicit/a c cent feedback /annotation (step 624) from the USfif, If. Can algP 
result 

in initia t e n CT a live connection with a remote human or automated agent which 

mav 

culminate in a commercial transaction- Purina the playback mode, it is 
preferred 

that, if the same media content is being indexed by the reading of multiple 
labels repetitious playback of the same content is avoided. 

Label identification in the playback mode is virtually the same as 

the 

label identification in the authoring mode. While label identification 
initiates 

object creation in the authoring mode, label identification initiates label 
matching followed by media rendering (if the label has an object identifier) 
in 

the playback mode. Furthermore, in playback mode, in addition to manual label 
reading, label reading may be automatically initiated either by a location- 
aware 

wireless network, an RFID tag in the proximity of the device, or by an 
internal 

clock trigger system. As noted, the outcome of the label identification 
process 

is an object identifier that can be used for indexing media content. 

[0075 ,ga& JL] Once a match is found in n Hnt- -.>>-» noi-h ft index table repository for 
the object 

identifier, media content bound to that object identifier can be sequentially 
rendered, provided that the media content is supported by the mobile personal 
device 207. A Playback of media content can be triggered in three ways, 
namely, by a 

user manually initiating the label identification, by the automatic reading 
of a 

label, or by a sequential presentation, e.g., a linear traversal of elements 
of 

a tour. o^ -Ref erring to fig. 2. the first two me^hedproactive methods 203 of 

triggering 



playback enable the tour to provide a user experience somewhat similar to 
having 

a human guide; the manual triggering being equivalent to the user asking a 
particular question and the automatic triggering £Q£jDeing equivalent to an 
ongoing commentary. Thus, the tour provides a richer user experience than the 
one provided by a human guide since these two methods of playback serve as 
two 

logical channels containing multiple media streams. To ensure that two 
channels 

do not rnnfliflt. and the transition between these two channels iff seamless, 

one 

channel can be designated as a background channel which has a lower rendering 
priority than the other. When a background feed is being inhibited as a 
function 

of its lower priority, an application may choose to provide a user with an 
interface cue (e.g., audio, graphics, text, or video) that indicates a 
background feed is available. FIG, 2 plota the ob-iect sizes 201 on the X axis 

[007 6 ] It io poooiblo during the label idontif ioation proocoo that a label 
detected 

and the Label detection range 202 on the Y axis. It illustrates that 
proactive 

lahel dete ction scheme works for small objects with low detection range and 
implicit label detection 204 works for large objects with longer detection 
range. Furthermore, as user moves between small and large objects with 

detectio n ranges, the transition between these domains 205 is made fleamlftflfl 

the background and foreground channel scheme as described above. The various 
label detection schemes that apply for these different domains are listed in 

the phyoical world doco not have a oorrooponding objoot identifier 

in a databaoo. In thio capo, the tour may bo authored to provide alternate 

index 

lookup oohemoo to find an unmatched index ouch ao, — for example, an index 
ooaroh 

in oelcct URLo. If the index io found, then that index can be added to the 
tour'o databaoo and the content can then become part of the ordered clcmonto 
e£- 

tho tour. 
PIG. 5b. 

f ^ ^0082 1 During the playback mode, generally illustrated in FIG. Sb-rl^ a 
user may be 

given the ability to annotate content as particularly illustrated as steps 
805 

and 806 in FIG. S-a-^-8 . The media for accepting annotations depends upon the 
capabilities of the device that accepts the annotations. When multiple 
objects 

qualify for annotation, a user should be prompted to choose among these 
multiple 

objects. An example of this may arise when a user stopped playback of a 
manually 

scanned object and the location of the object happens to coincide with a 
coordinate for which content is available. Feedback, illustrated in steps 807 
and 808 nf vm. 8 a. could aloo m av be made an interactive process. Still 
further, the tour may also 



support the notion of a live-agent connection facility which enables the user 
to 

connect directly to a human agent to initiate a transaction. This is 
particularly useful when the mobile personal device 2£3-is embodied in a 
cellular 

telephone. The user may initiate an electronic e-commerce transaction using 
the 

established connectio n - . During the tour the uocr may ocnd aoyiichronouo j -Jafefi- 
^onnection being made to a live or automated Want. 

mcooagco to other noern ^ nnmnnminifinn no.twork. Thio meooagc can do a 

voice mail meooagc left ^ -* nomnm -mnnnn prntnntGd voiQG mail boac picked up 

the rocipiont of the moooago from the mail box ("pootc rcotantc") . The 
meooagc 

can be a reminder nl^r*- nnnrW hnrnn.lf dnlivorcd at a future time. Tho 

oyotcm may apply trano format iono on the meooagc ouch ao, by way of example, 
converting a voiccmail to text and poot it on a wob oitc, or create an CMS 
meooagc, or email rcproocntation of tho meooagc and deliver it to the 
addrcooco . 

f ^ ^0083 1 As noted above, the authoring and playback of a tour imposes no 
constraints on the physical location of a tour or its contents, i.e., it 
could 

be locally resident on the mobile personal device or remotely resident on a 
server. When remotely located, the tour can be accessible by one of the 
several 

wireless access methods such as, for example , WPAN (Wirelcoo Pcroonal Area 
Network) , WLA N (Wirelcoo Local Area Network) , and wireless wide area network 
("WW AN (Wirelcoo Wide Area Network^ ) . Furthermore, the media content could be 
pre- fetched, downloaded on 

demand, streamed, etc. as is appropriate for the particular application. 

rno7Q 0084 1 Feedback and annotation provided in the context of a tour, the 
creation 

of which is generally depicted as 631 in FIG. 6 including steps 632-634, 
could 

also be resident in any physical location, in step 632, annotations and 

are arch wp 4 loca l ly ™ the mobile deyice 105 or unloaded to a server 114 
w^th 

trime and version informati on that heln identify fcheir creation times. Since 

feedback *n<i annotation mav be hard to interpret separate from the tour flue 
to a 

lack of context, annotati o n and feedback mav be merged 633 with the tQUr. 
Since 

feedback/annotation is bound to object identifiers that provide the context 
for 

the annotation/ feedback, it is also possible to create a tour subset of an 
original tour that contains only those elements which have annotation and 
feedback. This would be very useful if the user is interested not in 
recapitulating the entire tour but only those parts that were annotated or 
for 

which feedback was provided. To this end, a tour application running on a 
PDA, 

for example, can easily send the annotations and feedback to an appropriate 
destination as an email attachment for rendering by a party of interest as a 
new 



tour, m other f 0 rmi . hour publishing 634 with feedback and annotation CQUld 

unloading to a server. An example of this usage is a Parent annotating ft 
child's 

language l ^mino process, described in detail below. After thft Parent 

flMlPtfltSS ^ J J4 . . fcU *-v 

the tour, the tour mav be unloaded to a Berver 634 for sharing it with the 

rest 

rnnasi fi o. 9 illustrates usage of the Bvfltem in both a wired and wireless 
network f o r niavback of a tour. The stepfl listed here have be*n illustrated 

in 

detail i n ptgs. 6-8- Tf the device is not wireless network enabled (step 9Q1) 
then the tour is downloaded bv a wired connection (step 914) from the 

network. 

The next step is t o detect a label (step 902) . decode and normalize the label 
(step 90 3 )- and in the wireless network case (step 904). download the media 
from 

the remote server (step 915). Tf the device is not network enabled, content 
ifl 

retrieve d from local store (step 905) since it is has already been, 

ftrangfaraefl ^ ^ , t£ 

h v a wired connection. The content is then rendered (step 90S). If 

annotatio n /feedback is enabled (step 907), then for a PUbllC tQUX (fftftP 9Q8) , 
the annotation is unloaded (step 912) to server 913 if a connection (step 

910) 

is available. 

r0086l I f a connection is not available, it is Queued (step 911) for future 
unload. A nnotation for private tours are stored locally (SteP 909) , 

rnnftQ [00871 The following descriptio n and Table, with the aid o£ Tflbl e 3 1 and 
Tablo 2 set forth 

belowj. generally describe applications in which £ke& tour may be used. 

1 TABLE 1 Application categories Type Description of Application Labeling 
scheme 

1 Physical label-based applications barcode, RFID, IR, text strings, any 
label 

np^nnh t- r. i-mrh nhriTig n t^t; ca n be transformed to digital data bv some 
transduction means , times tamp 2 

Location-based applications Coordinates, t^ptd t digital text strings, opcooh - 
^ *-r»rt» Hh-r-ingn, any label 

that can be transformed to digit- al data by some transduc- tion means, 

timestamp 3 Timestamp based applications timestamp 4 Linear ordering based 
applications no label, application depends on linear ordering of tour 
content . 

[00 8 1] 
r00881 

2 TABLE 2 Examples of Applications Device Application Application Labeling 
Purpose Server # Name Description scheme Built PDA Phone Support 1 My First 
Child's voice cataloging T ime-stamp X Optional - Words cataloging w hile 
needed only if (Type 3) child is learning device hao to 



4- 



needed only if (Type 3) speak. Parent network can annotate connectivity 
-^^^flYlfifl T* afl child's utterances 

network CQ n n flfl hivihv. ffln ^fchorad bv a parent /child mav be UPlPflflfifl U fl 
aerver ufiintr an intermediate host, such a PC 2 Childs Childs label based 

learning . 

Hand-writton X Mn innrmnnrontent learning device, labclo device Objects in 
the (numbering) (Type 1) h ouse written authored bv a 

device are tagged or Barcode b y parent. Child labels parent/child (Type 1) 
identifies the 

distinctive 'n^Wl"^) mav be taqs on object and scans or Barege wlpflflftfl 
to a_ 

them to get an audio server using feedback. This device can an^also he used 
to 

scan intermediate a nnotated books with frost auch a PC embedded tago 3 
Travelers Label 

objects and record H and- written XXX Only for Language record name of 
x^o^oH-iefft in a foreign , 
written phone Learning r^jnnt in n f mm i an language labels Tool, (numbering) 
Tnni i -mrpnrtn (Type 1) o r Barcode (Type 1) 4 

Picture Album catalogin g , home H and- written XXX Only for album pbifictfi 
cataloging , homo labclo 

written phone annotation nHjnntn nnhninmno^ahala (Type 1) (numbering-) — (Typo 
i) or Barcode 5 Class Lecture W hen 

professor tmnr i T nHrt onuses a Hand- XXX Only for Annotation uoco a ftgcUrs 
printed labclo phono (Type 1) b ook as the ( numbering) wr i tten pfagne 
Annotation reference for his or Barcodo lectures, Iflbslg (TYPQ 1) his lecture 
can be 

spliced numbering) bu the student and he can or Pfrrcodg correlate the page 
of 

the book with the appropriate annotation from the lecturer. 6 Package Useful 
for 

managing a H andwritten XXX Only for Annotation, managingmpye, , a collectors 

labels phone Cataloging mnvn . .-if or cataloging possessions, (numbering) 
Private oollcctoro droam or 

Barcode Collectibles for cataloging (DVD, CD, pooooooiono. b ooks, etc) (Type 
1) 7 Focuo Wino taoting, Handwritten X 

X X Only for Croupo, product rating for labolo phono Marketing oonoumor 
roporto, 

(numbering) Information otc. or Barcode Collection, Product Rating Tool (Typo 
±) — 8— Shopping Record and 

playback B arcode. XXX Only for List playback grocery shopping list or 

Handwritten phone 

(Type 1) ohopping liot or labclo other to-do list 0 Pcroonallabels 8 Antique 
Seller labels Handwritten XXX Only for Retail objects, handwritten X 
x x Only for Shows, authors labclo phono Applicationo content, buyer 
(numbering) Art & Crafto p lays back or Barcode and Antique content 
r.hnwn , 1 ahftlg phone Auctions, frack 

content (n umbering) Art or BarrnHp Galleries car showroom - Art Callcrioo 
label parts of oar (Type 1) 

of car to explain features of the product 10 Networking Attondooo wear 
TTnTirWii-i-rm VQ City. Multimedia T mira of cities 

X Only for Tarty, device readable labolo, phono Singloo Party badgco, each 
Barcode, — (Type 1) pcroon can IR tago publioh a ohort introduction of 
hor/himoclf 



11 Talking Dirootiono, otorc Barcode, X X Only for Mallo, dircotory RFID 
phono 

Outloto, — information, 

Barcode X Y Por Phonp, MUB6 W and ffUgfiWW ffflflftHT RgiP Pftg flflY* <7? TftUKB. ftSt 

i ahft l. with »oh M o y y c«n fl ri e9 Coordinates Gtorco, ooupono, opooialo. Retail 
product 

rcvicwo, — (Typo 1 and price comparioon, Type 2) Guide to ohopping mallo, 
outloto, — 

retail otorco, etc. 12 Pootc The oorvioo offoro Primary label X X Yco 
Rcotantc a 

voice and web can be (Typo 1, accoooible location, Typo 2, Type pergonal 
barcode, etc. 3, nn^ Typ n nnimmiTi-i mfi rm nnnnndary 1) portal on a ocrvor 

label ! ^ 
for people to timootamp leave touro for othcro to uoc. 13 Talking Children 

can 

go Coordinatco X X Only TT-mninmn 1-rn.imirQ hunting and phyoioal phono 
Muocumo, in ocioncc contcro labclo Callcrico, and tho more (barcode, 
Exhibitiono, talking treaoureo RFID, lit, Trade they find and etc) Chowo, 
loam 

they are Geicnoc rewarded. Ccntcro Noto: Talking (Typo 1 and Treaoureo tour 



Type 2) not limited to audio, it may include any multimedia content 14 
Talking 

Tour of famouo Coordinatoo, X X Only for Gravco comet or ico RFID, phono (Typo 
■2- 

and (Arlington, Pore toxt otringo, Typo D LaChaioo opecch to — Ccmotcry, text 
Hollywood forever cemetery, etc) Find A Grave biographic touro of cclobritico 
(Graccland) 15 Talking National park Coordinatoo. X X Only for Trailo nature 
trailo. RFID, phono (Typo 2 and (Grand Canyon, text otringo, Typo 1) etc) 
opocch to — text 16 Talking Tour Cuidoo for Coordinatoo, X X Only for Citioo 
citico and RFID, phone (Type. 2 and buildingo, text otringo, Typo 1) Froodom 
Trail in opocch to — Dooton, The Mall text in Waohington D.C., intorioro of 
hiotoric buildingo churchoo, town hallo, hiotoric ohipo, etc 17 Voice Trailo 
Waypoint Coordinatoo X X Yoo (Typo 2) annotationo. People can oharc their 
oxpcrioncco, opiniono. Multiple authoro can author content for tho oamo 
label . 

The individual expcricnoco are aggregated on a web oito hooted on the 
internet 

into a oharod tour of the community. Authoro can upload to the tour hoot oito 
and uooro can download to their mobile apparatuo. Example all pcoplo who are 
walking the Appalachian Trail rocord their diary, qonjaectivjity (Types If 2, 
Tiiqe stamp r 

3 r and 4^ linear ordering 

r&ag3 -0089 1 Examples of applications are shown in Table 2, applications 1-9. 
For 

example, the system and method can be used for cataloging the early words of 
a 

child (Table 2, application 1). All parents can fondly recall at least one 
memory of their child's first utterance of a particular word/sentence. They 
are 

also painfully aware that it is so hard to capture those invaluable moments 
when 

the child makes those precious first utterances of a word/sentence (by the 
time 

parent runs off to fetch an audio/video recorder, the child's attention has 



shifted to something new and it is virtually impossible to get the child to 

sa y 

it again) . Also the charm of capturing the first utterance is never the same 
as 

the subsequent utterance of the same word/sentence. 

r&63 ^0090 1 To solve these problems, the apparatus described herein can be 
used to 

create a tour with a voice-activated recorder which records audio and 
catalogs 

it using a timestamp as the index. The system can be used to aggregate 
words/sentences spoken separately for each day thus serving as a chronicle of 
the child's learning process. The system can also be used to permit 
annotations 

of the authored content, the authored content being the child's voice. For 
example, a parent can annotate a particular word/sentence utterance of a 
child 

with the context in which it was uttered making the tour an invaluable 
chronicle 

of the child's language learning process. 

f^^4 -0Q911 The system can also be used to allow the parent to author multiple 
separate sentences in the parents own voice. This sentence would be randomly 
chosen and played when the child speaks to thereby encourage the child to 
speak 

more. The authored tour and the annotation can be retrieved from the device 
for 

g ,f.-^^ flT ^ for awing with others bv uploading to a rfimotft garvaff. 

Uploaded rpnfrent ma y be made a ^eseible afl public or Private tQUITB aCCeggifrlQ 
bv 

a nellular phone or PD A with wireless network COIfflfiCtivi tv . Though digital 
voice 

recorders of different flavors abound in the market, none of them match the 
key 

capabilities of the present invention which makes it best suited for this 
application. In particular, these devices do not support annotations of 
already 

recorded content nor authoring by a parent which is subsequently played as 
responses to the child speech which can serve to encourage the child to speak 
more . 

r&as ^0092 1 The above-described functionality of the system can be integrated 
into 

child monitoring devices existing in the market today, such as the "First 
Years" 

brand child monitor. Specifically the capability of this embodiment may be 
integrated into the transmitter component of the device. It will be 
appreciated 

that the receiver is not an ideal place for integration since it receives 
other 

ambient RF signals in addition to the signals transmitted by the transmitter. 

f ^€ ^0093 1 In still another application, the system and method can be used as 
a 

child's learning toy (Table 2, application 2). Preferably, in this 
application, 



a child-shield that selectively masks certain apparatus controls can be 
placed 

on the personal mobile devic e 207. A The "toy usage" of the apparatus 
highlights ease 

of content authoring and playback. In an example of this application, a 
mother 

labels objects in her home (or even labeling parts of a book) using barcode- 
or RFID labclOj __ 

m?TD or anv other 1 * Ha1 type that can be transduced bv SOfflft analog tQ digital 

F^ efma . and records information in her own voice about those objects. The 
child 

then scans the label and listens to the audio message recorded by the mother. 
The mother could hide the label in objects around the house, making the child 

9° 

in search of the labels, find them and listen to the mother's recording. It 
would thus serve the purpose of a treasure hunt. 

r.ftjw ^Q094 1 Yet another usage of the system and method is as a foreign 
language 

learning tool for an adult (Table 2, application 3). When an object is 
scanned, 

the personal mobile device would play the name of that object in a particular 
language. Still further, the system and method can be used to implement a 
digital audio player where the indexing serves as a play list. 

r-66a a0095 1 in its usage as a cataloging apparatus, the subject system and 
method can 

be used to catalog picture albums, books, CD, DVD oollcctiono, boxes during a 
move to a new 

apartment, etc. (Table 2, applications 4, 5-7—6). The system can rely on a 
simple 

labeling — h~ — r,nT " 1 v>n nnppi nmnnfnH with which could involve 

using labels that arp already present on the 

pre printed, oclf adhGoivo barcode labolo (oimilar to thooc uocd ao pootal 

nrirlr^nn l^nin) th i-h- i n T-ncpT-ri r n pb1 _ e .c.ts of interest or affixing custom 

^bels on the objects. . A user might 

label the pictures, etc. in any desired order with a unique number. 
Coincident 

with the labeling, or subsequent to the labeling process, the user may author 
content for a particular index and manually preserve the association between 
the 

index value of a picture, etc. and the authored content. Should the mobile 
personal device £0 ^105 include a barcode scanner, the barcode scanner can 
assist in 

maintaining the correspondence between the picture, etc. and the authored 
content by supporting coincident authoring of content with the label 
detection. 

In this implementation the labeling scheme would be done using any 

barcode -encoding scheme that can be recognized by the barcode reader. In this 

scenario the author of the tour and the playback of the tour might be the 

same 

person or different persons. 

r^&fr »Q096 1 The mobile personal device 3W12ji can also provide interface 
controls for 

providing digital text input, e.g., an ordinal position of content in a tour. 
It 



. * 



may have an optional display that displays the index of the current content 
selection. Interface controls can provide an accelerated navigation of 
displayed 

indices by a press -and-hold of index navigation buttons thus enabling the 
device 

to quiclcly reach a desired index. This is advantageous since the index value 
may 

be large making it cumbersome to select a large index in the absence of 
keyboard 

input. The mobile personal device ££3 -105 could also be adapted to remember 
the last 

accessed index when the device is powered down to increase the speed of 
access 

if the same tour is later continued. In further embodiments, the personal 
mobile 

device a ^lOS can have a mode selector that allows read only playback of 
content . 

This avoids accidental overwrite of recorded content . 

rfta^ s0097 l When the system and method is used as a "personal 
cataloger /language 

learning/audio player," then the tour authoring and playback apparatus 
need 

only be provided with object scanning capability as it is intended for 
sedentary 

usage and, therefore, need not support coordinate-based labeling. This 
personal 

mobile device £6^105 can be adapted to allow multiple tours to be authored 
and 

resident on the device at the same time. 

f .06 ^00981 The system and method can also serve as a memory apparatus, for 
example, 

assisting in the creation of a shopping list and tracking the objects 
purchased 

while shopping to thereby serve as an automated shopping checklist (Table 2, 
application %2) * To this end, the system can maintain a master list of object 
identifiers with a brief description of these objects created in the 
authoring 
mode . 

f -0-0 ^0099 1 Table 2, applications in 17 8 and 9 are examples of tours 
particularly targeted 

to cellular phones and handheld devices (PDA) . The system can be used as a 
tour 

authoring and playback device that implements all forms of object labeling 
and 

indexing mentioned earlier, e.g., text strings, gpocch - to toxttransduqed 
analog frn digital 

data , barcode, RFID, IR f location coordinate, and timestamp. All of the tours 
may include any multimedia content and are not limited to audio. One 
application 

of such a "tourist -guide" is a tourist landing at an airport and using the 
system to obtain information about locations, historical sites, and indoor 
Ob j ^.g - r flpamlesslv transitioning between proactive and implicit label 
detection 



Another application io a oightoccing walking tour (Table 2, application 16) 

of a , . . 

hiotoric town whore an outdoor otrcct tour io intermixed with vioitmg 

interiorci 

of buildingo along the way. In thio application, a variety of labeling 

mcthodo . 
may be uocd ao depicted on FIG. 5. It can be appreciated that multi lingual 
vcroiono of tho tour may be bound to the oamc labolo. It can bo appreciated 
that 

in a city where tho vir^^ ^ nmhin t-.n rnad atreot oiano due to language 
barrioro (ouch ao Weotcrnor cannot road Japanooc lottoro) , or a blind poroon, 
□ till would be ablo nimn -information ao oomconc proficient in 

tho 

local language. Another application of tho apparatuo io a uoor going to a 
largo 

ohopping mall, and uoing the apparatuo to navigate the mall, and to find 
information on itcmo in a otorc. 

[00Q3] "Pooto Rootanto" oorvicc (Table 2, application 12) offcro a voice and 
wob 

acceooiblc pergonal communication portal (multimedia mailbox) on a ocrvcr for 
people to leave tonro f " y ^ unn. Tho owner and authorized vioitoro 

aoceoo the pergonal portal (multimedia mailbox) via a toll free telophono 
number 

or T-ia a ucb browocr Th^ ^mmr nnn innvn ro.mindp.ro to horoolf (where did I 
parked my car?) or oharo touro (ouch ao "My Firot Wordo") with friendo and 
family or even otrangcro . 

[0001] In yet another application tho tour io built by multiple authoro and 
the 

tour repreoonto the oharod oxperiencco of a community (Table 2, application 
17) . 

Tho tour io a collection of annotated waypointo. The tour io hooted at an 
Intornct web oito. Authoro can upload label oontont pairo and add them to tho 
tour. Uoero can download tho tour to their mobile apparatuoco. Authoro and 
uooro 

can be the oamc or different pcroono . An example of ouch a tour can be hikero 

the Appalachian Trail that record location coordinate label and pergonal 
diary 

content pairo and upload the pairo to the tour'o web oito. Vioitoro of the 
web ■■ 

oitc in turn arc able to download the tour to their pergonal mobilo 
apparatuoco . 



[0005] By way of more opecific examplco, FIG . 1 illuotratco an embodiment of 
the 

mobile guide oyotcm where the application io a tour of a ohopping center. The 
figure illuotratco two aopecto of the oyotcm, namely, a method of mapping 
phyoical world locationo and objecto into digitally otorcd object identificro 
otorcd in a databaoc and the uoo of uniform object identificro for locationo, 
buildingo and individual objecto in the oamc oyotcm. The tour otarto with the 
vioitor approaching tho outlet center. Map 100 depict o the location and 
dircctiono to center 101 which can be prcocntcd to the uocr ao a rooult of 
reading a "label in the air." The object identifier for the outlet center io 
derived from ita location coordinatco. 



[000 6 ] Similar information can be prcocntcd to the uoor ao tho uoor navigatoo 
through the coordinatco within building 101 which containo upper level 102 
and 

lower level 103. Each level containo Dtorco. On lower level 103 there io 
otoro 

10 4 (Store 11 in the local directory) . Store 104 containo dreoo 105 that can 

labelcd with a unique barcode which the uocr can read to receive information 
about the dreoo. Thuo, tho vioitor can browoc thio phyoical world equipped 
with 

a handheld mobile device 207 and tho tour io a "booth in" from large otatic 
objecto to omall mobile objooto ao tho vioitor makoo her way from otrcct, to 
building, to floor, to otoro , finally to the dreoo. Tho larger otatic objeoto 
contain the omallcr mobile objooto. Thio containment property of opacoo and 
objecto aido tho oyotom in narrowing down the location of the vioitor inoido 
the 

building. For large otatic objecto ouoh ao otrccto and buildingo the oyotom 
dcrivco an object identifier from tho geographical pooition of tho object. 
Once 

tho vioitor turno her attention to omall mobile objecto ouoh ao a dreoo, then 
the longitude and latitude of tho vioitor io no longer relevant. Therefore 

oyotom dcrivoo tho object identifier for omall mobile objecto from machine 
readable tago, ouch ao commercial barcodoo. 

[0007] To facilitate tho tour, an oxamplc of the handheld devico can be an 
Bricooon GCM telephone model R520, R320, T20, etc. with a barcode ooannor 
attachment. In another example, the, ohopping center can be wired with 8 02.11 

Bluetooth Wirelcoo Local Area network (WLAN) and tho vioitor can uoc a PDA 
with 

a WLAN network interface card (NIC) to communicate with the local wirelcoo 
network. The oyotom can retrieve additional information about the vioitor 'o 
location ("label in tho air") by tracking which wirelcoo WLAN acccoo point 
tho 

vioitor 'o NIC oonnocto to and by approximating tho diotanoc of tho NIC from 
tho 

acccoo point baocd on RF oignal otrength. Additional information may bo 
generated to help to determine the NICo location by logging tho movement of 
the 

NIC uoing timcotampo and comparing the laot know pooition of the NIC with ito 
current approximated pooition. 

[000 8 ] In another opecific example, illuotratcd in FIG. 0, the application io 

guided tour of cemetery 000. Vioitoro walk along the road among tho gravco 

and try to find gravco of famouo people or loved onco. The labclo marking tho 
gravco trigger the playback of the content bound to that label, and the 
vioitor 

with the mobile device can hear the voice of the pcroon honored with tho tomb 
otonc, occ the pcroon'o image on the dioplay of a PDA, — etc. creating a 
opecial 

uocr experience. It can be appreciated that there io an intangible benefit 
when 



a plaao or an objoat (the tomb otono in thio gqog) , or a poroon long paoood, 
oan 

dirootly "talk" to the vioitor. It oan bo a muoh moro cathartic experience 
than 

a prpoentation by a "middle man" ouch ao a livo tour guido. 

[0000] Tho figure illuotratoo throo different dovicco with difforont 
oapabilitico uocd to take tho oamo tour. Tho three devioco arc ; — (1) cellular 
telephone with local CPS receiver , or network baocd GPS oorvor; — (2) PDA with 
WLAN or WWAN modem connection; and (3) PDA without network connection. In 
more 

dctailo, the firot vioitor uoco a cellular phono 002 equipped with a built - in 
CPS pooitioning receiver 003. Tho phono docodoo the GPS coordinatoo 
longitude/latitude and oendo the ooordinatoo through cellular baoc otation 
013 

to a remote oorvcr platform 01 8 . Server platform 01 8 rccoivoo the roquoot, 
tranoformo the location coordinatoo into an object identifier > looko up the 
content aooociated with tho object identifier, and oendo back tho information 
about nearby grave 001 to phono handoot 002. Alternatively the phono dooo not 
have built in GPS receiver, and inotcad it rctricvoo ito location from a 
remote 

location oorvor. Additionally the vioitor may oay the name of the pcroon on 

tomb and othor identifying information ouch ao date of birth or death. Tho 
oorvor oonvcrto opcoch to text and uooo the text otring ao label to look up 
tour 

information. Depending on tho capabilitioo of the phone, tho information can 

a voico rcoponoc or a dioplay of additional graphical information in a 
wirolcoo 

browoor that io running on the phono. Server platform 01 8 may oupport oomo or 
all of the following protooolot Voioo/IVR/VoiocXML, HTTP, WAP Gateway, SMS 
mcooaging, I Mode, GPRS, and other wirolooo data communication protocolo 
known 

in tho arto. 

[0100] A occond vioitor uooo a pookot PC 00 6 ouch ao, for example, a Compaq 
iPAQ, with dual communication oloto wherein plot 007 oontaino an RFID roador 
and 

plot 00 8 houoco cither a 8 02.11 WLAN Network Interface card (NIC) or a 
Bluetooth 

NIC. A nearby grave 004 hao RFID tag 005 mounted on it. RFID reader 007 rcado 
RFID tag 005, and tranoformo the RFID tag information to a univeroal object 
identifier. Alternatively if the PDA doco not have an RFID reader, the 
vioitor 

may enter the name on the grave ao a label. Pocket PC 006 connccto to a 
Wirclcoo 

Local Area Network (WLAN) Accooo Point Old uoing a WLAN NIC (Network 
Interface 



Card) — 008. Wirclcoo Acoeoo 


point 014 connccto through local area notwork 015 
oorvcr platform 016. Alternatively, the WLAN NIC 


local content diotribution 
can 

be oubotitutcd with a CDPD 


wirclcoo modem card or other WAN network card that 



enablca the PDA to connect to a cellular data network. 



[0101] A third vioitor uooo a Handopring Vioor 012 with a Springboard module 



w * 



nriD reader 911- A n^ar^y gv^rn O OP hao RFID tag DIP mounted on it. RFID 

reader , 
Oil rcado nriD tag 910 ani +-y--mnf nrmn t-ho rftd hag information to a univcroal 
object identifier. Ao an alternative to RFID, the vioitor can enter the name 



xm — , 

the grave ao label. Yin?* nrm 010 dnpn nQt navG a network connection. It 
otorco 

object identifiero and content locally on the device. ■ 

rmmi Trr-nT nH nmaing 2 <^ Fnrthennnre. from the foregoing, it will be 

appreciated that the 

described system and method bridges the world of object -based information 
retrieval and location-based information retrieval to thereby provide a 
seamless 

transition between these two application domains. 

roiOOl In particular, the described system provides, among others, the 
following 

advantages not found in prior systems: 

rA3-fta -01011 (1) Using the Internet as an easily accessible vast information 
resource , 

off-the-shelf multi-media capable portable handheld devices and ubiquitous 
wireless networks, the present innovation provides an open, interactive guide 
system. The user is an active, interactive participant of the guided tour, a 
creator and supplier as much as he/she is a consumer. Applications are only 
limited by imagination- -ranging from educational toy, trcaoure hunt in a 



r,r,-, r,nnr ri1t | in ^ l i r . . 

center, bargain hunt in a ohopping mall, touring hiotono citico or famouo 
ccTtictcrico, attending nnhunriring pnrHnn where people wear machine readable 
i^a^ ^^^incr tours etc. In all of these applications, the user, with the 
aid of the 

present invention, is able to personalize, annotate the tour with his/her own 
impressions, share feedback with other users, initiate an interaction or 
transaction with other humans or machines. 

[ 010-1 01021 a. The individual may create hio/hcr own object tago, and label 
^TMf^-p ^h^ 861 ^** Qr ua * thft existing labels on 
objects around her. 

f.ft4 ^0103 1 b. The author of a tour and the user of a tour (supplier and 
consumer) 

might be the same person (s) or different person (s) . 

f4W ^0104 1 c. A "private tour" can be easily published to the Internet or to 
a local 

community, and made "public" for other people to use, contribute, exchange or 
sell. 

rm07 0105 l d. The tour is no longer a closed, finished product, --it can be 
personalized, shared, co-authored by people who have never met in person 

r6±£ #oiQ6 1 e. Users may use their personal portable handheld devices, instead 
of 

renting specialized proprietary devices from institutions, and download only 
the 

software and content from the internet or local area networks. 



f 0l00 0107 1 f . Users and service providers have access to authoring tools to 
author 

and publish multimedia content including streaming video and audio. 

[ QHQfl lS&l 9- The system provides system and method, to author and publish a 
tour, 

but the system does not restrict the content of the tour. 

[0111] — (2) Prior oyotcmo treat location baocd oorvicoo and object labeling ao 
two ocparato tochniquoo . The current invention trcato thooc two aopocto of 
the 

phyoical world ao labeled objeoto of different ocaloo. Small mobile objecto 
and 

largo otatic objecto (ouch ao buildingo a.k.a. locationo) — arc both modeled 
with 

the oamo data otructurc, and ao labeled objecto. The current invention can 
naturally accommodate phyoical objecto of all ocaloo, — and rolationohipo among 
plurality of phyoical objecto around uo. 

[ OH 2 . £3Lfl£] (32.) Tn e system can be used both indoors and outdoors. 

r oil3 0110 1 (4-i) Tour content can be authored in different media types. The 
tour 

presentation depends on the capabilities of the device (audio only, text 
only, 

hypertext, multimedia, streaming video and audio etc) and would do 
appropriate 

media transformations and filtering. A tour would work both with and without 
network access. The user can download the tour content before the tour, and 
store it on a portable handheld device, or access the tour content 
dynamically 

via a wireless network. 

r oii4 01in The system takes advantage of both existing object tags 

(barcodes, 

RFID, Infrared tags) and specialized tags made for a specific tour. 

f 0ii5 01121 The benefit of the logical aggregation of related content 

into a tour 

is clearly apparent, not just in the multitude of commercial applications, 
but 

also in the multitude of personal usage scenarios, such as an audio annotated 
album, a chronological repository of a child's early utterances, or a tour 
containing a mothers' annotation of her old home and the articles she left 
behind bequeathed to her children. The tour serves, in these cases, as an 
invaluable time warp triggering recall of fond memories that enrich our 
lives . 

It also plays the important role of immortalizing humans with a media rich 
snapshot of their lives. 

fQ1131 Although the invention has been particularly shown and described with 
reference to several preferred embodiments thereof, it will be understood by 

[0116] — It will bo appreciated by those skilled in the art that various 
changes in form and details may be made 

modif icationo and alternatives to the opocific embodimento doocribed could be 
developed in light of the overall tcachingo of the diooloourc. Accordingly, 
the 



1 



particular arrangement diocloocd io meant to bo illuotrativo only and not 
i^^^CI " n * n ^ nt - hftT ; flin without departing froro the spirit and scope of the 
invent i- n T 7 n+-h^. 1 " Vtn ^ nirnnHnT1 ^ n bn givnn aa defined 
the full breadth ofja tne appended claims and any cquivalcnto thereof . 
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April 22, 2002 



RECEIVED 

APR 2 5 2002 

BROBECK 



Re: U.S. Patent Application Serial No. 10/035,952 

Title: SYSTEM AND METHOD FOR AUTHORING AND PROVIDING 

INFORMATION RELEVANT TO A PHYSICAL WORLD 

Our File No.: 66566.01 US2 ■ 

Dear Mr. Tate: 

Enclosed please find a copy of the above-referenced patent application and a 
copy of the Combined Declaration for the signature of Mr. Ajit Rajasekharan. Once 
Mr.Rajasekharan has signed the Declaration, I kindly ask that the Declaration be 
returned to my attention for filing with the United States Patent Office. 

The subject patent application was filed on December 26, 2001 and claims 
priority to U.S. provisional patent application Serial No. 60/306,356. As you are aware, 
the provisional patent application correctly names both Rozsa Kovesdi and Ajit 
Rajasekharan as inventors. 

It is our understanding that Mr. Rajasekharan has filed a patent application that 
also claims priority to this provisional patent application. It is our further understanding 
that Mr. Rajasekharan has represented to the United States Patent Office that he is the 
"sole" inventor of the subject matter recited in the claims of said patent application. 
However, we call your attention to the fact that Mr. Rajasekharan has admitted on 
several occasions that Ms. Kovesdi must be considered to be an inventor of the subject 
matter that is recited in the claims of the provisional patent application. For example, in 
a correspondence dated August 31 , 2001 , Mr. Rajasekharan stated: 
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"The system patent - the one we already filed. There 
is no doubt in my mind about who its inventors are - 
Rozi and Ajit in equal measure." 

"The overall device patent - not yet filed, I am 
supposed to have long completed it and I have not. 
This is the subsidiary patent to the system patent and 
shares the same descriptions as the system patent - 
though Ajit may have a larger contribution in this 
patent as Rozi herself persistently claims, the 
evolution of ideas were a consequence of our constant 
combined thinking. So I would state that there was 
equal contribution to that too." 

"in fact, even with both our contributions to the 
system and device patent - there have been seyeral 
instances where I would say some statement which may 
not have a direct relevance to a claim but it would 
trigger the right idea in you and vice versa. Most of 
the claims in our system are the results of such 
confluences of both our ideas - to me it is a futile 
exercise to dissect them apart and say who contributed 
what." 

Since you have informed us that the claims set forth in the patent application filed by Mr. 
Rajasekharan are identical to those found in the provisional patent application and, as 
such, identical to those in the subject patent application, we request that Mr. 
Rajasekharan either: 1) add Rozsa Kovesdi as an inventor in the patent application that 
he has filed; or 2) withdraw his previously filed patent application in favor of the subject 
patent application. 

We believe that these corrective measures are required to protect the property 
interests of Ms. Kovesdi and Mr. Rajasekharan under the Patent Law while also 
providing an amicable resolution to this matter. 

If you have any questions, please call me. 
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via Facsimile 
(202) 778-2201 

Rodger L. Tate, Esq. 
Hunton & Williams 
1900 K Street, N.W. 
Washington, D.C 20006-1109 



Re: U.S. Patent Application No. 10/035,952 

Entitled: SYSTEM AND METHOD FOR AUTHORING 

AND PROVIDING INFORMATION RELEVANT 

TO A PHYSICAL WORLD 
Your Reference No.: 66566.01US2 



Dear Mr. Tate: 

Please be advised that we have removed ourselves as counsel from the above-referenced 
matter All further correspondence and documents should be forwarded directly to Ms. Kovesdi. 



Very truly yours, 
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Enclosed please find a copy of the above-referenced patent application and a 
copy of the Combined Declaration ft r the signature of Mr. Ajit Rajasekharan. Once 
MrRajasekharan has signed the Declaration, I kindly ask that the Declaration be 
returned to my attention for filing wffl i the United States Patent Office. 

The subject patent application was filed on December 26, 2001 and claims 
priority to U.S. provisional patent ap >lication Serial No. 60/306,356. As you are aware, 
the provisional patent application cojTectly names both Rozsa Kovesdi and Ajit 
Rajasekharan as inventors. 

It is our understanding that Mk Rajasekharan has fifed a patent application that 
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We believe that these corrective measures are required to protect the property 
interests of Ms. Kovesdi and Mr. Rajasekharan under the Patent Law while also 
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September 16, 2003 



RODGER L. TATE 
DIRECT DIAL: 202-419-2069 
EMAIL: rUte@hunton.com 



VIA FACSIMILE & FEDEX 



FILE NO: 63044.5 



Rattan Nath ' 
Pennie & Edmonds LLP 
1 155 Avenue of the Americas 
New York, N.Y. 10036-2711 

Re: U.S. Patent Application No. 10/035,952 ("the *952 application") 
Filed: December 26, 2001 

Entitled: SYSTEM AND METHOD FOR AUTHORING AND PROVIDING 
INFORMATION RELEVANT TO A PHYSICAL WORLD 

Your Ref.: 11326-003 

OurRef.: 63044.5 : 

Dear Mr. Nath: 

We acknowledge receipt of your facsimile in the late afternoon of September 15, 2003, and 
your proposed response enclosed therein to the outstanding Office Action dated March 18, 
2003, in the above-referenced patent application. Due to the belated nature of your attempt to 
provide a response in accordance with M.P.E.P. § 402.10 and the imminent deadline of 
September 18, 2003, we find that a reasonable amount of time does not exist for us to properly 
consider the response and to provide you with the necessary changes to assure that Mr. 
Rajasekharan's full interest in this application is preserved. 

We note that our proposed response prepared on both parties' behalf was forwarded to you on 
May 21, 2003, shortly after you originally notified us of the outstanding Office Action. This 
response was provided to you at an early stage in the statutory period for responding, i.e., 
nearly four months prior to the now looming deadline, in order to give you plenty of time to 
provide us with your and Ms. Kovesdi's comments. Until yesterday, we had not received any 
further communication from you in regard to this matter. Because yesterday's letter does 
nothing more than to merely acknowledge the receipt of our proposed response, it appears that 
Ms. Kovesdi has chosen to completely disregard Mr. Rajasekharan's timely attempt to advance 
the formulation of an appropriate response suitable to both parties. 

As stated in the United States Patent & Trademark Office's Decision on September 17, 2002 
(Paper No. 12), to assure that all interests are properly and effectively represented, all further 
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correspondence to the PTO must be signed by both a representative of Mr. Rajasekharan and a 
representative of Ms. Kovesdi. 

We will not risk harming Mr. Rajasekharan' s interest in the above-referenced application by 
hurriedly executing and submitting your "eleventh hour" response. If you wish to preserve Ms. 
Kovesdi's interest in the application, please file the amendment as originally presented in our 
proposed response forwarded to you on May 21, 2003. 
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D ETAILED ACTION 
Response to Amendment 

! 

1 . Acknowledgement is made of ijeceipt of Response filed on September 18, 2003 by the 
first attc rney, Mr. Rattan Nath from P^nnie & Edmonds LLP. 

2. Acknowledgement is made of i eceipt of supplemental Response filed on September 22, 
2003 by the first attorney, Mr. Rattan ' Nlath from Pennie & Edmonds LLP, concerning the 
communication between Mr. Rattan Nath and Mr. Rodger L. Tate from Huton & Williams. 

3. The reply filed on September 1 8, 2003 is not fully responsive to the prior Office Action 
because of the following onrission(s) <j>r matters): The Amendment filed on September 18, 2003 
lacks signature from an inventor's representative Mr. Rodger L. Tate, or from Hunton & 
Williams. The supplemental response filed on September 22, 2003 is not considered as a 
signature from Mr. Rodger L. Tate because the supplemental response only shows a 
correspondence between the two representatives, Mr. Rattan Nath from Perinie & Edmonds LLP 
and Mr. P^odger L. Tate from Hunton & Williams, and does not represent a signature for the 
Amendment filed on September 18, 2003. Appropriate signatures from both representatives are 
required for consideration of Amendment. See MPEP § 402.10 and 403 as well as 37 CFR 1.33. 
See 37 CFR 1.111. Since the above-: nentioned reply appears to be bona fide, applicant is given 
ONE (1) MONTH or THIRTY (30; i DAYS from the mailing date of this notice, whichever is 
longer, within which to supply the or lission or correction in order to avoid abandonment. 
EXTENSIONS OF THIS TIME PElilOD MAY BE GRANTED UNDER 37 CFR 1 136(a). 



i 
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4. The following is a courtesy copy of the previous Office Action mailed March 18, 
2003. 



5. The numbering of claims is not 
original numbering of the claims to be 
canceled, the remaining claims must 
must be numbered consecutively beg 
numbered claims previously presented 
Misnumbered claim 23, 24, 2p 
70 respectively. 



Claim Objections 

in accordance with 37 CFR 1 . 1 26 which requires the 
preserved throughout the prosecution. When claims are 
be renumbered. When new claims are presented, they 
with the number next following the highest 
(whether entered or not). 
..69, 70, 71 have been renumbered 22, 23, 24. ..68, 69, 



not 



nming 



6. A rejection is based on double 
the language of 3*5 U.S.C. 101 which 
useful process ... may obtain a patent 
invention," in this context, means an i 
Eagle Mfg. Co., 151 U.S. 186(1894); 
and In re Vogel, All F.2d 438, 164 



A statutory type (35 U.S.C. 101) 
amending the conflicting claims so 
terminal disclaimer cannot overcome 



7. Claims 1-31, 33-42 and 45-70 
the same invention as that of claiml-7|0 
provisional double patenting rejection 



Double Patenting 

patenting of the "same invention" type finds its support in 
states that "whoever invents or discovers any new and 
therefor ..." (Emphasis added). Thus, the term "same 
invention drawn to identical subject matter. See Miller v. 
In re Ockert, 245 F.2d 467, 1 14 USPQ 330 (CCPA 1957); 
UJiPQ 619 (CCPA 1970). 



double patenting rejection can be overcome by canceling or 
thoy are no longer coextensive in scope. The filing of a 
double patenting rejection based upon 35 U.S.C. 101. 



are provisionally rejected under 35 U.S.C. 101 as claiming 

of copending Application No. 09/987597. This is a 
since the conflicting claims have not in fact been patented. 
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i Rejections -55 USC §102 

8. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the 
basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(b) the invention was patented or described in a printed publication in this or a foreign country or in public use or on 
sale in this country, more than one year prior to the date of application for patent in the United States. 

9. Claims 18, 19, 21, 22, 31, 34,48, 54 and 58 are rejected under 35 U.S.C 102(b) as being 
anticipated by Liu (US 5,480,306, as cited by the Applicant). 

Liu teaches a method and apparatus for providing information relevant to a physical 
world by reading a bar code associated with a sound data and the code is converted into a 
memory address pointer pointing to the initial address of the memory area in which the digital 
sound or pronunciation is stored and t le sound applied to a loud speaker system (col 2 lines 
35+). Liu teaches that the bar code is printed on visible media, such as paper and plastic slides. 
The conversion from the bar code to t le digital code then to an address pointer is considered be 
normalizing a read object label associated with object into an object identifier as discussed 
above. And placing the object identifi vc into an index table repository and binding the content to 
the object identifier are taught in Fig !>A and Fig 5B. 

Re claim 1 9: Liu teaches that the language learning apparatus contains a digital sound 
data memory means (col 6 lines 30-32). 

Re claim 54: Liu teaches that the memory control means 5 may properly retrieve the 
desired digital speed data of the word|from the memory means 6, which inherently shows that 
there is not pattern or sequential ordeir for accessing, therefore it is randomly accessible. 

Re claim 58: The apparatus is a purpose build device targeted to read bar code. 
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Claim 

10. The following is a quotation of 
obviousness rejections set forth in this 



Rejections -35 USC §103 
35 U.S.C. 103(a) which forms the basis for all 
Office action: 



(a) A patent may not be obtained thoug i the invention is not identically disclosed or described as set forth in 
section 102 of this titie, if the differences between the subject matter sought to be patented and the prior art are 
such that the sub'ict matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to whicl l said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was ma ie. 



11. Claims 1-5, 9, 13, 16, 29, 30, 39-42, 59, 60, 63-64 and 66-67 are rejected under 35 U.S.C. 
103(a) as being unpatentable over Liu (US 5,480,306) in view of Savchenko et al (US 
6,111,567). 

Liu teaches a method and apparatus for reading a bar code associated with a sound data 
and the code is converted into a memcry address pointer pointing to the initial address of the 
memory area in which the digital sour d or pronunciation is stored, converted to an analog signal, 
and the sound applied to a loud speak* system (col 2 lines 35+). The apparatus 10 is considered 
to be a circuitry. Liu teaches that the bar code is printed on visible media, such as paper and 
plastic slides. Liu teaches thart: the apparatus having a memory and a speaker means for 
outputting the sound (col 6 lines 22-5 )). 

Liu fails to teach a method fbo authoring information and a system for authoring the 

content. 

Savchenko teaches methods of authoring multimedia titles (col 1 lines 8-10). 
Therefore, it would have been obvious to an artisan of ordinary skill in the art at the time 
the invention was made to integrate i ie teachings of Savchenko to the teachings of Liu in order 



1 
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to create a well organized system so 
provide a good quality sound and 

Re claim 2 and 3: Liu fails to 



that 



minimal memory is utilized, but at the same time 
maintain the flow of the music or sound produced. 

that the system for authoring content is resident in the 



teach 1 



apparatus: 

Savchenko teaches that the execution instructions for the authoring tool are contained in 
the memory (col 4 lines 39-47). 

Therefore, it would have been \ )bvious to an artisan of ordinary skill in the art at the time 
the invention was made to integrate th ; teachings of Savchenko to the teachings of Liu in order 
to provide a multifunctional apparatus so that the user may author and playback the sound 
according to his/her preference utilizir g only one apparatus, which avoids complicated 
connections between multiple devices. 

Re claim 9: Liu fails to teach t lat the step of storing the content in non- volatile memory 
resident in the apparatus. 

Savchenko teaches that a computer application 42 is stored in the non- volatile memory 
34 (col 4 lines 37-47). 

Therefore, it would have been obvious to an artisan of ordinary skill in the art at the time 
the invention was made to integrate tfte teachings of Savchenko to the teachings of Liu in order 
to safely store the content so that the fontent is not easily changed or modified by others. 

Re claim 13 : Savchenko furtto * teaches a computer readable storage media having 
instructions for authoring informatior (col 4 lines 39-47). 

Therefore, it would have been obvious to an artisan of ordinary skill in the art at the time 
the invention was made to integrate the teachings of Savchenko to the teachings of Liu in order 
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to speed up the process by storing all tfie instruction in the memory and have the processor 
access and execute the instructions instead of loading or inputting the instructions one-by-one by 
the user. 

Re claim 16, 29 and 30: In addition to Liu as modified by Savchenko discussed above, 
the conversion from the bar code to th .5 digital code then to an address pointer is considered be 
normalizing a read object label associated with object into an object identifier as discussed 
above. And placing the object identifux into an index table repository and binding the content to 
the object identifier are taught in Fig 5 A and Fig 5B. 

Re claim 42: Liu teaches that rendering digital multimedia as a function of output 
capabilities of the apparatus (col 6 lints 50-58). 

Liu fails to teach programming that renders digital multimedia as a function of output 
capabilities. 

Savchenko teaches a computer application 42 that executes instructions (col 4 lines 36- 



45). 



Therefore, it would have been 



i obvious to an artisan of ordinary skill in the art at the time 
the invention was made to integrate tlie teachings of Savchenko to the teachings of Liu in order 
to provide the proper sound signal tha t matches the output characteristics of the apparatus so that 
the user can listen to a good quality sound with less background noise and interruption. 

Re claim 63: Liu teaches that 1 keyboard for inputting information (col 1 lines 13-25). 
12. Claims 6 and 23 are rejected mder 35 U.S.C. 103(a) as being unpatentable over Liu as 
modified by Savchenko as applied to claim 1 above and Liu as applied to claim 18, and further in 
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view of Cave (US 5,958,014). Liu as modified by Savchenko and Liu have been discussed 
above. 

► ind Liu fail to teach that the content is a link to a live 



Liu as modified by Savchenko 



agent. 



Cave teaches device having auc io capabilities and can be connected to a live agent (col 1 
lines 65+). 

Therefore, it would have been obvious to an artisan of ordinary skill in the art at the time 
the invention was made to integrate the teachings of Cave to the teachings of Liu as modified by 
Savchenko in order to provide a two-way audio or text exchange to communicate with each other 
without remembering or dialing numbers, which also makes the process faster. 
13. Claims 7, 14, 36, 37, 49 and 53 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Liu as modified by Savchenko as applied to claim 1 and 13 above and Liu as applied to 
claim 38, and further in view of Conle; r, Jr. et al (US 6,434,745). 

Liu teaches receiving a plurality of optical codes (col 7 line 45). 

Therefore, it would have been < >bvious to utilize the steps of Liu as modified by 
Savchenko and repeat steps for as mar y coded labels necessary because it is a mere duplication 
of process. 

Liu as modified by Savchenko 
entity called a tour. 

Conley teaches that a tour component of the browser 8 allows the end-user to identify on? 
or more URLs and save them into a group called a tour and to create one or more such tours, and 



fails to teach aggregating the content into a single logical 
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to save each tour to a searchable local tour database on the end-user computer 14 similar to the 
searchable local image database. 

Therefore, it would have been c bvious to an artisan of ordinary skill in the art at the time 
the invention was made to integrate th4 teachings of Conley to the teachings of Liu as modified 
by Savchenko in order to organize the (lata so that related data are grouped in the same group. 
Such modification helps and speeds up; the searching process when the data needs to retrieved 
because the data are 

14. Claims 8, 15, 33 and 35 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Liu as modified by Savchenko as applied to claim 1 and 13 above and Liu as applied to claim 31 
above, and further in view of Brooks et al (US 4,963,719). 



Liu as modified by Savchenko 



i fails to teach detecting a second label associated with the 
first object and normalizing the first la yc\ and the second label such that the content bound to the 
first object can rendered during detection of either the first or second label in the playback mode. 

Brooks teaches two labels associated with the same object, two labels attached to an 
object and detecting two of the labels i Fig 2, col 2 lines 26+). 

Therefore, it would have been obvious to an artisan of ordinary skill in the art at the time 
the invention was made to integrate the teachings of Brooks to the teachings of Liu as modified 
by Savchenko so that plurality of bar <;ode labels having the same sound or data may be provided 
on different or multiple appliances fori duplication or convenience purposes. 
15. Claim 10, 11, 12, 20, 25, 45-47 and 65 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Liu as modified by jSavchenko as applied to claim 1 and 64 above and over 
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Re claim 10, 20, 45 and 65: Liu 
uploading and downloading the conten 



Cluts teaches a remote server 34 utilized to transmit programming information for storage 



by one or more of the memory storage 



Cluts teaches a communicatioc 
Therefore, it would have been 



as modified by Savchenko and Liu fail to teach a step of 
to a remote server. 



! devices 3 0 (col 1 4-27) » 
Therefore, it would have been obvious to an artisan of ordinary skill in the art at the time 
the invention was made to integrate the teachings of Cluts to the teachings of Liu as modified by 
Savchenko because a remote server m: ty provide more memory or storage space, which allows 
more data and information to be storec . 

Re claim 1 1 and 47: Liu as mo iified by Savchenko fails to teach that the step of 
uploading is performed via a wireless letwork. 

link is wireless (col 7 lines 14-27). 
i obvious to an artisan of ordinary skill in the art at the time 
the invention was made to integrate the teachings of Cluts to the teachings of Liu as modified by 
Savchenko because it does not require wired connection, therefore provides mobility and 
convenience. 

Re claim 12 and 46: Liu as mc dified by Savchenko fails to teach that the step of 
uploading is performed via a wired network. 

Cluts teaches a communication link is wired (col 7 lines 14-27). 

Therefore, it would have been obvious to an artisan of ordinary skill in the art at the time 
the invention was made to integrate the teachings of Cluts to the teachings of Liu as modified by 
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Savchenko because the possibility of <hta loss or corruption in transferred data decreases, 
therefore the modification provides a more accurate transmission of the data 

Re claim 25: Liu fails to teach ^hat the step of rendering the content comprises streaming 
the content from a remote server. 

Savchenko teaches rendering the content comprise streaming the content (col 1 lines 28- 

31). 

Therefore, it would have been obvious to an artisan of ordinary skill in the art at the time 
the invention was made to integrate the teachings of Savchenko to the teachings of Liu in order, 
to transmit the sound file through communication links using industry standards, such as MPEG 
standards. 

fails to teach a remote server. 

! 4 utilized to transmit programming information for storage 
devices 30 (col 14-27). 



Liu as modified by Savchenko 
Chits teaches a remote server 2 
by one or more of the memory storage 



Therefore, it would have been obvious to an artisan of ordinary skill in the art at the time 
the invention was made to integrate the teachings of Chits to the teachings of Liu as modified by 
Savchenko because a remote server rr ay provide more memory or storage space, which allows 
more data and information to be store I 

16. Claims 17, 62 and 70 are rejected under 35 U.S.C. 103(a) as being unpatentable over Liu 
as modified by Savchenko as applied to claims 16, 59 and 64 above, and further in view of 
Bridgelall (US 6,264, 106). Liu as modified by Savchenko have been discussed above. 

Liu as modified by Savchenkc fail to disclose that the instructions allow a plurality of 

i 

different label types to be normalized to one object identifier. 
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provide a data that corresponds to the 
timestamp corresponds to each other. 
18. Claims 26 and 27 are rejected 



Bridgelall teaches a combinatio|i bar code scanner/RFID circuit for reading bar code or 

RFID (col 2 lines 20+). 

Therefore, it would have been cjbvious to an artisan of ordinary skill in the art at the time 
the invention was made to integrate the teachings of Bridgelall to the teachings of Liu as 
modified by Savchenko because bar cede and RFID tags are commonly known forms of 
identification and combining those twej functions into one device will provide the flexibility of 
reading different types of codes. 
17. Claim 24 is rejected under 35 llj.S.C 103(a) as being unpatentable over Liu as applied to 
claim 18 above, and further in view of Fan et al (US 6,324,165). Liu has been discussed above. 

Liu fails to teach determining t le current time and comparing the current time to the 
timestamp before rendering the content. 

Fan teaches a timer issuing a current time and a comparator for comparing the queue 
timestamp to the current time (col 27 lines 7-12). 

Therefore, it would have been pbvious to an artisan of ordinary skill in the art at the time 
the invention was made to integrate the teachings of Fan to the teachings of Liu in order to 



current time by checking to see if the current time and 



under 35 U.S.C. 103(a) as being unpatentable over Liu as 
applied to claim 18 above, and furthei in view of Boulton et al (US 5,566,291). Liu has been 
discussed above. 

Liu further teaches that the language learning apparatus contains a digital sound data 
memory means (col 6 lines 30-32). 
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Liu fails to teach that the steps of accepting annotations/feedback after the rendering of 
the content and binding the annotations 'feedback to the object identifier. 

Boulton teaches an object identifier field 200 that stores an object identifier which 
references an object the user may be referencing with his or her feedback information. In 
Boulton's feedback system, objects can be used to further define the context when the feedback 
is provided (col 25 lines 55+). 

Therefore, it would have been obvious to an artisan of ordinary skill in the art at the time 
the invention was made to integrate the teachings of Boulton to the teachings of Liu in order to 
enhance the content and provide a better quality sound, image, etc. to the user by editing or 
making additional comments to the content. 

19. Claim 28 is rejected under 35 U.S.C. 103(a) as being unpatentable over Liu as modified 
by Boulton as applied to claim 26 above, and further in view of Cluts. - 

Liu as modified by Boulton far s to disclose the step of storing the annotations/feeback in 
a remote memory. 

Cluts teaches a remote server 34 utilized to transmit programming information for storage 
by one or more of the memory storage devices 30 (col 14-27). 

Therefore, it would have been pbvious to an artisan of ordinary skill in the art at the time 
the invention was made to integrate this teachings of Cluts to the teachings of Liu as modified by 
Boulton because a remote server may provide more memory or storage space, which allows 
more data and information to be stored 
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20. Claim 32 is rejected under 35 Uj.S.C, 103(a) as being unpatentable over Liu as applied to 
claim 3 1 above, and fiirther in view of ^wartz et al (US 6,095,4 1 8). Liu has been discussed 

i 

above. 

Liu fails to teach that at least 01 Le of the plurality of labels is custom created. 

Swartz teaches translating the MIDI code to a symbol data and to music print data. The 
printer 26 then prints the symbol data £ s symbol 14. 

Therefore, it would have been obvious to an artisan of ordinary skill in the art at the time 
the invention was made to integrate th< : teachings of Swartz to the teachings of Liu in order to 
provide custom created bar code so that information regarding the content's location within the 
database or other information may be e ncoded according to the program or application that is 
used for the system 

2 1 . Claims 43 and 44 are rejected i inder 35 U.S.C. 1 03(a) as being unpatentable over Liu as 
modified by Savchenko as applied to c laim 38 above, and further in view of Hollander (US 

Liu as modified by Savchenko have been discussed above, 
fail to teach that the physical world comprises labeled 
locations containing labeled mobile objects and the labeled locations are used to determine 
proximity of the labeled mobile objecis. 

Holland teaches labled locatio is, such as lobeled bin or labled shelf (col 4 lines 48-51). 
Therefore, it would have been obvious to an artisan of ordinary skill in the art at the time 
the invention was made to integrate the teachings of Holland to the teachings of Liu as modified 
by Savchenko in order to identify the purpose, use or the physical description of the location of 
the labeled location so that the locatiojns can easily and quickly identified. 



4,037,302) and Blum (US 4,654,727). 
Liu as modified by Savchenko 
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Blum teaches that a bar code lajjel on the cassette is read by a bar code reader in order to 

i 

enable a computer control system to determine the location of the cassette and control the 
subsequent transport of cassettes to the! tape transports (col 1 lines 40-45). 

Therefore, it would have been bbvious to an artisan of ordinary skill in the art at the time 
the invention was made to integrate the teachings of Blum to the teachings of Liu as modified by 
Savchsnko in order to quickly and easily determine the location of the object by using the bar 
code label as a tracking method, whict also avoids the object from getting lost. 

22. Claim 48 is rejected under 35 I J S C. 103(a) as being unpatentable over Liu as modified 
by Savchenko and Cluts as applied to claim 47 above, and further in view of Aguirre et al (US 
6, 195,53 1). Liu/Savchenko/Cluts have been discussed above. 

Liu/Savchenko/Cluts fails to teach that the wireless network comprises a cellular 
telephone network 

Aguirre teaches a cellular telep hone network (col 3 lines 26-40). 

Therefore, it would have been Dbvious to an artisan of ordinary skill in the art at the time 
the invention was made to integrate the teachings of Aguirre to the teachings of 
Liu/Savchenko/Cluts because it is a widely used wireless network method for providing a safe 
and reliable data transmission, which isnhances the accuracy of the data being transmitted. 

23. Claims 50 and 51 are rejected under 35 U.S.C. 103(a) as being unpatentable over Liu as 
applied to claim 38 above, and further in view of Cole et al (US 6,359,71 1). Liu have been 
discussed above. 

Liu fails to disclose that the apparatus accesses the tour via the internet and a voice 

portal. 
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Cole teaches a portable compute with access methods of voicemail and internet (col 2 
lines 10-15). I 

Therefore, it would have been dbvious to an artisan of ordinary skill in the art at the time 
the invention was made to integrate the teachings of Cole to the teachings of Liu because it is a 
fast method to communicate and transr lit data using a conventional phone line, which is 
commonly available. 

24. Claim 52 is rejected under 35 IJ.S.C. 103(a) as being unpatentable over Liu as applied to 
claim 38 above, and further in view of Aguirre and Cole. 

Liu fails to teach that the apparatus accesses the tour via a cellular telephone voice, 
mailbox. 

Aguirre teaches a cellular telephone network (col 3 lines 26-40). 

Therefore, it would have been obvious to an artisan of ordinary skill in the art at the time 
the invention was made to integrate th 5 teachings of Aguirre to the teachings of 
Liu/Savchenko/Cluts because it is a w dely used wireless network method for providing a safe 
and reliable data transmission, which <mhances the accuracy of the data being transmitted. 

Cole teaches a portable computer with access methods of voicemail and internet (col 2 
lines 10-15). 

Therefore, it would have been 3bvious to an artisan of ordinary skill in the art at the time 
the invention was made to integrate thje teachings of Cole to the teachings of Liu because it is a 
fast method to communicate and transmit data using a conventional phone line, which is 
commonly available. 
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25. Claim 55 is rejected under 35 U 



f S.C. 103(a) as being unpatentable over Liu as applied to 
claim 38 above, and further in view of itrueger (US 5,598,540). Liu has been discussed above. 
Liu fails to teach that the digital! multimedia is accessible by the apparatus in a sequential 

order. 

i 

Krueger teaches accessing the s ored data only in sequential order (Abstract). 

Therefore, it would have been obvious to an aitisan of ordinary skill in the art at the time 
the invention was made to integrate the teachings of Krueger to the teachings of Liu in case 
presentation of the digital multimedia r lust be in a certain order for the user to understand certain 
topics before achieving the next data so that the next data makes more sense to the user. 
26. Claims 56 and 57 are rejected dnder 35 U.S.C. 103(a) as being unpatentable over Liu as^ 
applied to claim 38 above, and further n view of Ramachandran (US 6,315,195). Liu has been 
discussed above. 

Liu fails to teach that the appar itus comprises a personal digital assistant and a cellular 
telephone. 

Ramachandran teaches a portable terminal 14 that reads bar codes and also may be 
integrated into a carrier 62, which may be a personal digital assistant or a cellular phone (col 8 
lines 23-26, col 9 lines 35-42). 

Therefore, it would have been < obvious to an artisan of ordinary skill in the art at the time 

teachings of Ramachandran to the teachings of Liu 
and cellular phones have the capability of storing 
information and also wirelessly transmitting information through internet and other 
communication methods, which enhar ces the voice data and play back as well. 



the invention was made to integrate th i 
because both personal digital assistant 
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27. Claims 61 and 68 are rejected ui|der 35 U.S.C. 103(a) as being unpatentable over Liu as 
modified by Savchenko as applied to cl^im 59 and 64 above, and further in view of Chen et al 
(US 5,869,820). Liu as modified by SaVchenko have been discussed above. 

Liu as modified by Savchenko fails to teach that the circuitry comprises an IR tag reader. 
Chen teaches an infrared tag reader (col 8 line 5). 

Therefore, it would have been obvious to an artisan of ordinary skill in the art at the time 
the invention was made to integrate the teachings of Chen to the teachings of Liu as modified by 
Savchenko because IR tags are readily available tags that are also used for identification 
purposes and it utilizes wireless communication, which provides mobility and faster process. 

28. Claim 69 is rejected under 35 U.S.C. 103(a) as being unpatentable over Liu as modified 
by Savchenko as applied to claim 64 above, and further in view of Bertram et al (US 5,613,137). 
Liu as modified by Savchenko have be zn discussed above. 

Liu as modified by Savchenko fails to teach a circuitry determining a coordinate location. 

Bertram teaches a coordinate d itermining circuitry 302 configured to determine 
corresponding locations of the touch o i the coordinate sensor (col 15 lines 20-28). 

Therefore, it would have been obvious to an artisan of ordinary skill in the art at the time 
the invention was made to integrate th ; teachings of Bertram to the teachings of Liu as modified 
by Savchenko in order to determine th : location of the touch pad sensor input that provides the 
information regarding the identificatioh of the content to be retrieved and played back. 
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Conclusion 



5,971,279, discloses a handheld scanner for the visually 



29. The prior art made of record ancj not relied upon is considered pertinent to applicant's 
disclosure. 

Savchenko et aL, U.S. Patent N6. 6,343,298, discloses a seamless multimedia branching. 
Kikuda, U.S. Patent No 4,952,7: 55, discloses a bar code generating apparatus for image 
communication terminal device. 

Raistrick et aL, U.S. Patent No. 
impaired. 

Citron et aL, U.S. Patent No. 5,i88,976, discloses a bar code use in information, 
transactional and other system and service applications. 

Kunizawa et aL, U.S. Patent Nol. 4,964,167, discloses an apparatus for generating 
synthesized voice from text. 

Knowles discloses a hand-held (portable www access terminal with visual display panel 
an gui-based www browser program integrated with bar code symbol reader in a hand- 
supportable housing. 

Hoda et aL , U. S . Patent No. 4,831,610, discloses method and apparatus for interactive 
control of a data recording medium pk yback apparatus using bar code access. 
Barton et aL, U.S. Patent No. 5,998,752, discloses a sorting system 
Dyko et al, U.S. Patent No. 5,956,708, discloses an integration of link generation cross- 
author user navigation, and reuse identification in authoring process. 



V 
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Wilz, Sr. et aL, U.S Patent No. 
information-related transactions over 
with reading java- applet encoded bar 



examiner should be directed to Kumikc 
The examiner can normally be reached 
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,992,752, discloses an internet-based system for enabling 
th|e internet using java-enabled internet terminals provided 
symbols. 



code 



Any inquiry concerning this communication or earlier communications from the 



C. Koyama whose telephone number is 703-305-5425. 
on Monday-Friday 7am-3 :30pm. 
If attempts to reach the examin* r by telephone are unsuccessful, the examiner's 
supervisor, Michael G. Lee can be reached on 703-305-3503. The fax phone numbers for the 
organization where this application or proceeding is assigned are 703-308-7722 for regular 

After Final communications. 

or relating to the status of this application or proceeding 
hose telephone number is 703-308-0956. 



communications and 703-308-7722 foi 
Any inquiry of a general nature 
should be directed to the receptionist m 



Kumiko C. Koyama 
December 03, 2003 
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In re Application of 
Kovesdi and Rajasekharan J 
Application No. 10/035,952 
Filed: 26 December, 2001 
Attorney Docket No. 66566. 01US2 



DECISION DISMISSING PETITION 



JLP-. . ^ OIK*/**' 

This is a- decision on the petition J"*™ " ^j* 1 '' 2 °° 4 ' ^ 
is treated as a renewed petition under 37 CFR l.lbZ. 

THe 'petition is DISMISSED. 

on 26 December, 2001, the above-identified application was filed. 
Sapers riled on 23 August, 2002, in response to the Decision 
Under 37 CFR 1.47(a) and 1.182 mailed on 16 August, 2002, 
included a Declaration signed by the Previously non-signing 
inventor, Rajasekharan, in compliance with 37 CFR 1.63. 

A Petition under 37 CFR 1.182 requesting that all correspondence 
be required to be signed by representatives of both- joint _ 
inventors was filed on 23 August, 2002, and was granted on 17 
September, 2002. 

in the present renewed petition, petitioner Kovesdi argues, that. 
Petitioner Rajasekharan has refused to sign a response to the 
P Z ^hT™ office action. Petitioner Kovesdi requests that the 
"quiremen? 1 correspondence be signed by representatives 

or Doth joint inventors be withdrawn so that Kovesdi may, by 
herseU a response to the Office action. Petitioners also 

request that the petition under 37 CFR 1.47(a) be granted 
Petitioners' arguments have been carefully considered but are 
not Persuasive that the relief can or should, be granted. 

At the outset, it is noted that the petition, under 37 CFR 1.47(a) 
origSany fi ed on 13 May,- 2002, was dismissed as moot on 17. 
September! 2002, because both joint inventors have signed the 
deflation in compliance with 37 CFR 1.63 As such, Rule 47. 
status is not applicable to the present application. 
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Petitioners' argument that Rule 47 status should be granted 
because Ra jasekharan, after signing the declaration, refuses to 
cooperate in signing correspondence in the Office misinterprets 
the purpose of Rule 47. 

With regard to the request that the requirement that 
correspondence be signed by representatives of both inventors be 
revoked, petitioners' argument has been considered, but is not 
persuasive. When a petition under 37 CFR 1.182 requiring the 
signature of representatives of more than one applicant on all 
correspondence is granted, petitioners assume the risk that not 
all of the applicants may cooperate in signing correspondence. 
To grant this petition would essentially render the grant of such 
a petition under 37 CFR 1.182 of no effect, as petitioner could 
by such means file Office correspondence upon behalf of the 
recalc^rant inventor, to the prejudice of that inventor's 
rights It is noted that the general policy of the USPTO is that 
the owner of less than the entire interest of a given application 
will not be permitted to unilaterally control prosecution unless 
and until it is clearly shown that the rights of that party are 
being prejudiced by the action (s) or inaction (s) of the party (s) 
that constitute (s) the remaining interest. Nevertheless the 
mission of the USPTO is to accept and examine applications for 
patent that ultimately promote the progress of science and the 
useful arts, and a party constituting less than the entire 
interest who nevertheless seeks to bring an invention before the 
public will not, ultimately be thwarted in that endeavor. 

Petitioner Kovesdi may wish to consider filing a continuation or 
divisional application covering only the matter invented by her, 
naming herself as the sole inventor. In doing so, petitioner 
Kovesdi may prosecute the application without the necessity of 
obtain^g the cooperation of Rajasekharan. Should petitioner wish 
to pro^cute claims to the jointly invented subject matter as 
well, tnen petitioner may wish to consider filing a continuing 
application' and again seek relief under 37 CFR 1.47 if 
Rajase^aran does not execute the declaration. If however 
Rajasei-aran does join in that application and again displays the 
recall --ance herein complained of in failing to assisting in the 
prosec'-ion of that application containing the joint claims, 
petitioner may re-present the instant petition under 37 CFR 1.182 
and then seek the requested relief. 

With regard to petitioner's request for suspension of action, 37 
CFR"l.i)3(a) provides that: 
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On request of the applicant, the Office may grant a 
suspension of action by the Office . . . for good and 
sufficient cause. The Office will not suspend action if a 
reply by the applicant to an Office action is outstanding. 
Any petition for suspension of action . . . must specify a 
period of suspension not exceeding six months. Any petition 
for suspension of action ... must also include: (1) A 
showing of good and sufficient cause for suspension of 
action; and (2) the fee set forth in §1. 17(h), unless such 
cause is the fault of the Office. 

The above-identified application was filed on 26 December, 2001. 
On 16 December, 2003, the Office mailed applicant a non-final 
Office action. This action set a three-month shortened statutory 
period for reply, to 17 March, 2004, with requests for extensions 
of time under 37 CFR 1.136(a) permitted. A proper. reply has yet 
to be received. 

The pc-L.it ion must be dismissed as improper. Action cannot be 
suspended in an application awaiting a reply by the applicant. 

Further, applicant is specifically advised that the original 
three-month period for reply has expired, but that period may be 
extended by a request for such accompanied by the appropriate fee 
for extension of time, and the reply itself. The time for 
submitting a proper reply cannot be extended beyond 16 June, 2004 
(with payment of a three-month extension of time fee) . Failure 
to timely file a proper reply (including the appropriate 
extension of time fee) will result in the abandonment of the 
application . 

Based on the foregoing, the petition is dismissed . 

Counsel's deposit account, No. 50-3013, will be charged a fee of 
$130.00 for consideration of the present petition. 

Further correspondence may be addressed to the following: 

By mail: Mail Stop Petition 

Commissioner for Patents 

P.O. Box 1450 

Alexandria, VA 22313-1450 

By FAX : "0 3) 872-9306 

Attn: Office of Petitions 
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By hand: 



U.S. Patent and Trademark Office 
2011 South Clark Place 
Customer Window 

Crystal Plaza 2, Lobby, Room.lB03 
Arlington, VA 22202 



OR- 



U.S. Patent and Trademark Office 
220 20 th Street S. 

Customer Window, Mail Stop Petition 
Crystal Plaza 2, Lobby, Room 1B03 
Arlington, VA 22202 



Telephone inquiries related to this decision should be directed 
to the undersigned at 703-308-6918. 



Douglas I.. Wood 

Senior Petitions Attorney 

Officv. of Petitions 



rec.ive z- J\.::e, 2004, street addresses for several of the buildings of the 
United States Patent and Trademark Office (USPTO) , including the Crystal Plaza Two 
buildir.^ hich is the location of the Customer Window, have been changed. This change 
is mads :■■ :ause the sireet on which the Crystal Plaza Two building is located will be 
rede si =:::.:: ™a frcr. Sc'j-.n Clark Place to 20" fc Street 3. 




